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Abstract. We derive several identities that feature irreducible characters of the gen¬ 
eral linear, the symplectic, the orthogonal, and the special orthogonal groups. All the 
identities feature characters that are indexed by shapes that are “nearly” rectangular, 
by which we mean that the shapes are rectangles except for one row or column that 
might be shorter than the others. As applications we prove new results in plane par¬ 
titions and tableaux enumeration, including new refinements of the Bender-Knuth and 
MacMahon (ex-)conjectures. 


1. Introduction. We prove identities that set into relation irreducible characters 
of classical Lie groups. All of them feature a classical group character indexed by a 
“nearly” rectangular shape. What is remarkable about these identities is that they 
have simple explicit forms (as opposed to many, however more general, identities in 
the literature), and that many of them exhibit multiplicity-free expansions, typically 
featuring sums of characters indexed by shapes that have a hxed number of rows or 
columns of odd length. 

There are three types of identities that we consider. 

First, in Theorem 1, we express (irreducible) characters of the general linear group 
GL(A) (these characters are also known as Schur functions) of “nearly” rectangular 
shape in terms of (irreducible) characters of the symplectic group Sp(A), and in terms 
of (irreducible) characters of the orthogonal group 0(A). As may be expected, for 
the proofs of these identities we utilize Littlewood’s [25] branching rules for restricting 
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representations of the general linear gronps to representations of the symplectic or 
orthogonal gronps. 

Secondly, in Theorem 2, we express (irredncible) characters of the symplectic gronp 
Sp(A^) (where N = 2n) and (irredncible) characters of the special orthogonal gronp 
SO(iV) (and its spin covering gronp) of “nearly” rectangnlar shape in terms of (ir¬ 
redncible) characters of the general linear gronp GL(A^). For the proofs of these 
identities we nse the tableanx descriptions for symplectic characters developed by 
DeConcini and Procesi [3, 4], and for special orthogonal characters as developed by 
Lakshmibai, Mnsili and Seshadri [22, 23, 21], 

Finally, in Theorem 3, we express the prodnct of two (irredncible) characters of the 
symplectic gronp Sp(2n), respectively of the special orthogonal gronp SO(A^) (and its 
spin covering gronp), of “nearly” rectangnlar shape in terms of (irredncible) characters 
of the same type. For the proofs of these identitites, we rely on Littelmann’s extension 
[24] of the Littlewood-Richardson rnle for all classical gronp characters, which also 
nses the tableanx by DeConcini, Procesi, Lakshmibai, Mnsili and Seshadri. In fact, 
as a hrst “approximation”, we convert Littelmann’s rnle in the special case of the 
prodnct of an (irredncible) symplectic or special orthogonal character of rectangular 
shape by an arbitrary (irredncible) character of the same type to a simpler and more 
explicit form, the coefficients in the expansion being expressed in terms of modihed 
Littlewood-Richardson coefficients. The resnlting formnlas are given in Proposition 1 
in Section 6. 

We want to emphasize that it is particnlarly the tableanx descriptions by DeConci¬ 
ni, Procesi, Lakshmibai, Mnsili and Seshadri that tnrn ont to be very nsefnl and effec¬ 
tive here. The more classical rnles (cf. [12, 13, 16]) involving Littlewood-Richardson 
coefficients, which wonld also apply to the second and third problem type that we 
consider, are nice in theory bnt nseless in practice, since it seems to be impossible to 
keep track of the cancellations that are cansed by application of modihcation rnles. 

As will become apparent, the reason that classical gronp characters indexed by 
“nearly” rectangnlar shapes allow particnlarly nice identities is becanse Littlewood- 
Richardson hllings and the above mentioned tableanx behave special for “nearly” 
rectangnlar shapes. This fact was previonsly observed by Proctor and Stanley [28, 
29, 31] for rectangnlar shapes. Aside from these three papers, the inspiration for the 
cnrrent paper comes from [20]. There it was discovered that symplectic characters of 
“nearly” rectangnlar shape have a nice explicit expansion in terms of general linear 
characters. This gave the idea to explore what other identities for classical gronp 
characters of “nearly” rectangnlar shapes exist. 

Finally, we remark that Okada’s paper [27] may be considered as a precnrsor to 
this paper as it contains the specializations to rectangnlar shapes of all onr identities. 
Okada, however, nses a completely different approach (namely, he nses the minor 
snmmation formnla of Ishikawa and Wakayama [9]). It mnst be pointed ont on the 
other hand, that there is one type of identities that Okada adresses, bnt I am not 
able to address. This is restrictions of representations of G{N) {G can be GL, Sp, or 
SO) to a representation of G{1) x G{N — /). Thongh compntational evidence exists 
that the respective identities of Okada can be extended from rectangnlar shapes to 
“nearly” rectangnlar shapes as well, so far I was not able to prove anything becanse of 
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the lack of an efficient combinatorial rnle. (Again, the rnles in [12, 16] are apparently 
nseless.) 

Onr paper is organized as follows. In Section 2 we recall the dehnitions of ir- 
redncible general linear, symplectic, orthogonal, and special orthogonal characters. 
Then, in Section 3, we state onr resnlts. These are snbsequently proved in Sections 4, 
5, and 6. Section 7 contains applications of onr identities to plane partition theory, 
inclnding new rehnements of the Bender-Knnth and MacMahon (ex-)conjectnres. 
Finally, in the Appendix we provide the necessary backgronnd information, in par- 
ticnlar there we describe all the different types of tableanx that we nse, Littlewood- 
Richardson Filings, the Littlewood-Richardson rnle, Littelmann’s extension of it to 
other classical gronps, and Littlewood’s branching rnles for restricting representations 
of the general linear gronps to representations of the symplectic or orthogonal gronps. 

2. General linear characters (Schnr functions), symplectic, orthogonal, and 
special orthogonal characters. Here we recall the classical character formulas. We 
refer the reader to [5; 6, ch. 24; 13; 15; 33, Appendix; 39] for surveys and more detailed 
background information concerning classical group characters. 

In what follows x = (xi, X 2 , ■ ■ ■) will always be an inhnite sequence of indetermi- 
nates. We call a sequence A = (Ai, A 2 ,..., A^.) with Ai > A 2 > ■ ■ ■ > A^ > 0 a partition 
if all the A^s are integers^ and a half-partition if all the A^’s are half-integers, by which 
we mean numbers of the form /c + 1/2, where k is an integer. For (ordinary) parti¬ 
tions we adopt the convention, that partitions that only differ by trailing zeros are 
considered to be the same. The components Ai, A 2 ,... of A are also called the parts 
of A. We call a sequence A = (Ai, A 2 ,..., A^) with Ai > A 2 > ■ ■ ■ > A^-i > |Ar| an 
r-orthogonal partition if all the A^’s are integers^ and an r-orthogonal half-partition if 
all the Xfs are half-integers. 

Given a partition A = (Ai, A 2 ,..., Ar), we dehne the (formal) general linear char¬ 
acter, also called Schur function, s(A;x) by (see [26, I, (3.4)]) 

det (hA,-i+j(x)), (2.1) 

where hmi^X) = X]i<q<...<i^ denotes the complete homogeneous symmetric 

function of degree m. If r < n, 

s{\-,xi,X 2 ,... ,Xn,0,0,. ■.) 

is the irreducible character for GL(n, C) indexed by A (see e.g. [6, (24.10)]). Following 
King [13] we write 


Sn(A; x) := s(A; Xi,X 2 ,..., Xn, 0, 0,...). (2.2) 

Two further similar notations for Schnr functions that we use are 


S2n(A;x=^^) 

s(A; xi,x^ X 2 , ^2 . 

■ • 5 Xn, Xn 50,0,...) 

(2.3) 

S2n-ri(A;x=^^) 

:= s{X-, Xi, xf^, X 2 , xf^,.. 

. , ,1,0,0,...). 

(2,4) 


and 
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Again, let A = (Ai, A 2 ,..., A^) be a partition. Following Koike and Terada [15, 
Def. 2.1.1], we define the (formal) symplectic character sp(A;x) by 

sp(A;x)= det (/iA,-i+i(x) : hx,-i+j{^) + hx^-i-j+ 2 i^))- (2.5) 

l<i,j <r 

Here, the notation of the determinant means that the first expression gives the entries 
of the first colnmn and the second the entries for the remaining colnmns, j > 2. If 
r < n, 

Sp{X-,Xi,X^^,X2,X2^, . . . ,Xn,X~^,0,0, . . .) 

is the irreducible character for Sp(2n, C) indexed by A (see [6, Prop. 24.22]). We write 
SP2ni>^: := sp{>^: Xi, ^2, X2^,...,Xn,x~^,0,0,...). (2.6) 

If r < n + 1, 

sp(A; Xi, X^^, X2, X 2 \ ...,Xn, X~^, 1,0,...) 

is a character for Proctor’s odd symplectic group SSp(2n + 1,C) (see [30, Prop. 3.1 
with xjv = !])• We write 

«1^2n+l(-^;x^^) := sp(\; Xi, X^\ X 2 , X 2 \ ..., Xn, X~\ 1, 0, 0, . . (2.7) 

Following Koike and Terada [15, Def. 2.1.1], we define the (formal) orthogonal 
character o(A; x) by 


o(A;x) 


det (/iAi-i+i(x) - hAi-i-j(x)). 


( 2 . 8 ) 


If r < n. 


o ( A , X 2,^2 ,^ 2,^2 : • • • : , 0 , 0 ,...) 


is the irreducible character for 0(2n, C) indexed by A (see [6, Ex. 24.46]), for which 
we write 

02 n(A; x=^^) := o(A; xi, x^^,X2, x ^^,..., x~^, 0, 0,...). (2.9) 


Similarly, if r < n. 


0{X-,Xi,X-^ ^,X2,X2^, ■■■ ,Xn,X^^, 1,0,0, .. .) 


is the irreducible character for 0(2n + 1,C) indexed by A (see [6, Ex. 24.46]), for 
which we write 


02 n+l(A;X=^^) := o{X-,Xi,X^ ^ ,X 2 ,X 2 ^, ... ,Xn,X^^ ,1,0,0,...). (2.10) 

The character 02 n+i(A; x=*=^) is also the irreducible character for SO(2n + 1, C) in¬ 
dexed by A. Therefore we shall sometimes write S 02 n +i(A;x±i) fQj. 02n+i(A;x±i). 
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However, for the spin covering gronp of SO(2n + 1, C) there are also irredncible char¬ 
acters indexed by half-partitions. Let A = (Ai,..., A^) be a half-partition, then the 
irredncible character so 2 n+i(A;is given by the Weyl formnla (see [ 6 , (24.28)]) 


■5^^271+1 (A, X 


±u 


det 


det 


— (Ai+n—i+1/2) 


-{n-i+1/2) 

j 


( 2 . 11 ) 


In fact (see again [ 6 , (24.28)]), if A is an ordinary partition, then so 2 n +i(A;x±i) jg 
given by the same formnla. 

The sitnation is even more delicate for SO(2n, C). The irredncible characters for 
SO(2n, C) and its spin covering gronp are indexed by n-orthogonal partitions or half¬ 
partitions. Let A = (Ai, A 2 ,..., An) be an n-orthogonal partition or half-partition, 
i.e., Ai > A 2 > ■ ■ ■ > An-i > |An|, with the Ads integers, respectively half-integers. 
Then the irredncible special orthogonal character so 2 n(A;x^^) is given by the Weyl 
formnla (see [ 6 , (24.40)]) 


so2n(A;x=^^) = 


det 




+ T, 


) -F det (x^ 


^“(Ai+7i— pN 


— T ■ 


det [x^ " + T 


( 2 . 12 ) 

The irredncible character for 0(2n, C) indexed by the partition A, 02n(A; x=*=^), eqnals 
the irredncible character for SO(2n, C) indexed by A, so 2 n(A;x^^), if An = 0, bnt 
splits into two irredncible characters for SO(2n, C) if An 7 ^ 0, one indexed by A = 
(Ai, A2,..., An), the other indexed by A“, which by dehnition is (Ai, A2,..., An-i, 
—An) (see [ 6 , hrst paragraph on p. 411]). 


The characters s(A;x), sp(A;x), o(A; x) are also called universal characters, mean¬ 
ing that by specializing one obtains the actnal (general linear, symplectic, orthogonal, 
respectively) characters for any dimension of the corresponding gronp. There is no 
snch thing in the even special orthogonal case and for odd special orthogonal char¬ 
acters indexed by half-partitions. Of conrse, it has to be mentioned that there is not 
only a Weyl formnla for soiv(A;x=*=^), bnt also for Sn(A;x) (see [ 6 , p. 403, (A.4); 26, 
I, (3.1)]) and sp 2 n('^i^) (^^e [ 6 , (24.18)]), the latter reading 




det 


™-(Ai+n-i+l) 


det 




(2.13) 


Actnally, the Weyl formnlas for all these characters allow a nniform statement (see 
[ 6 , Theorem 24.2; 39, Theorem 4.1]). 

3. Character identities. In this section we collect onr identities for classical gronp 
characters of “nearly” rectangnlar shapes. By “nearly” rectangnlar shape we mean 
partitions of the form (c, c,..., c, c — p) or (c,..., c, c — 1 ,..., c — 1 ), i.e., partitions 



6 


C. KRATTENTHALER 


whose Ferrers diagram (see Section Al of the Appendix) is a rectangle, except that 
one row or colu mn might be shorter. We write — p) for the hrst partition 

(given that the partition has r components) and [c — 1)^) for the second (given 

that the partition has r components, p of which are eqnal to c — 1). 

The hrst hve identities express general linear characters (Schnr fnnctions) in terms 
of symplectic, respectively orthogonal characters. Most of the “partition terminology” 
that we nse here and henceforth is explained in Section Al of the Appendix. Other 
terminology concerns the parity of rows or colnmns: When we say ‘an odd (even) row’ 
we mean ‘a row of odd (even) length.’ We nse the same convention with colnmns. 
For convenience, we denote the nnmber of odd rows of some (ordinary or skew) shape 
a by oddrows(a) and the nnmber of odd colnmns of a by oddcols(cr), and the same 
with ‘even’ instead of ‘odd.’ 

Theorem 1. Let r, c, p by nonnegative integers. For c > p there holds 

s((c^“\c-p);x) = sp{iy]yi), (3.1) 

oddcols =p 


and for r > p there holds 

s((c’^“P, (c-l)P);x) = Y o(z/;x). (3.2) 

oddrows((c’')/!^) =p 


In particular, for r < N and c> p there hold 


sn {{ c ^ \c-p);x^^) = I 


oddcols(^(c^)/;^^ =p 


r < \N/2\ 


E r>\N/2\, 

i, oddcols(^(c^“’'+l =c—p 


(3.3) 


and 


r < n 


S2n+i((c’');x±^) = 


For p < r < [N/ 2J there holds 


i^C(c’') 

E r>n. 

i/C(c2" + l-^) 


(3.4) 




i/C(c’') 

oddrows^(c’')/!^) =p 


(3.5) 
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Remark. The identities (3.3)-(3.5) have obvious interpretations as branching rules 
for the restriction of representation modules of GL(A^, C) to Sp(A^, C), Sp(A^ — 1, C), 
respectively 0{N, C). 

The case N = 2n, r < n of (3.3) is implicit in Proctor’s paper [28, Proof of 
Lemma 4, Claim on p. 558]. In the same paper, there appear, explicitly, the case 
N = 2n, r < n, p = 0 of (3.3) [28, Lemma 4, equation for A 2 n-i{jn^^r)] and the case 
r < n of (3.4) [28, Lemma 4, equation for A 2 nijn^r)]- The cases N = 2n, r < n, 
p = 0 of (3.3)-(3.5) appear in [27, Theorem 2.6]. 

The next four identities express symplectic and special orthogonal characters in 
terms of general linear characters (Schur functions). 

Theorem 2. Let n^c^p by nonnegative integers. For n>p there holds 

SP2n{ic^~^Ac-ff)':^'^^) = {xiX2---Xn)~''- ^ Sn(z^;x). (3.6) 

i/C((2c)") 
oddrows (ly') =p 

If c is a nonnegative integer or half-integer and p a nonnegative integer, 2c > p, 
then there holds 

S02n ((c'"“^c-p);x=^^) = (a:ia;2 ■ ■ ■Tn)”''■ ^ Sn(z^;x). (3.7) 

!^C((2c)") 

oddcols^((2c)")/!/^ =p 

Next, if c is a nonnegative integer or half-integer and p a nonnegative integer, 
n> p, then there holds 

S 02 n+l{ic'^~^,{c-lY);X^^) = {xiX 2 ---Xny ■ ^ (z^) ■ (i^; x) , (3.8) 

i/C((2c)") 


where 


cin,p{^) = number of vertical strips of length p on the 

rim of n avoiding the {2c)-th column. (3.9) 

Finally, if c is a nonnegative integer or half-integer and p a nonnegative integer, 
c> p, then there holds 


S02n+l{{c^ ^,c-p);x^^) = {xiX 2 ---Xn) ^ bn,p{l') ■ Sn{l';x), 


!^C((2c)") 


where 


(3,10) 


bn,p{^) = number of horizontal strips of length p on the rim of v 
such that the i-th cell of the strip (counted from left to 
right) comes before the (2c — 2p + 2i)-th column. 


(3.11) 
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Remark. The identities (3.6)~(3.8), and (3.10) have obvions interpretations as decom¬ 
position formnlas for representation modnles of Sp(2n, C), SO(2n, C), or SO(2n-|-l, C) 
as representations of the snbgronp GL(n, C). For the interested reader we add that, 
withont mnch additional effort, it is also possible to derive decomposition formnlas 
for sp 2 n ((c""”^, c — p); and S 02 n (c — which are in the spirit of 

(3.10), respectively (3.8), by slightly modifying the argnments in the proofs of (3.10) 
and (3.8). 

Formnla (3.6) for the hrst time appeared in [20], althongh it is implicit already in 
[7, Theorem 2.6; 18, (2.2)]. The case p = 0 of (3.6) appeared already in a nnmber of 
papers [31, Theorem 3; 37, Theorem 4.1; 38, Cor. 7.4.(b); 27, Theorem 2.3.(2)]. 

The case p = 0 of (3.8) and (3.10), which is 

S02n+1 = ixiX2- ■ ■ Xn^ ■ ^ Sn(z^;x), (3.12) 

i/C((2c)") 

appeared also in a nnmber of papers [31, Theorem 3; 38, Cor. 7.4.(a); 27, Theo¬ 
rem 2.3.(1)]. 

Finally, the cases p = 0 and p = c of (3.7) previonsly appeared in [1; 27, Theo¬ 
rem 2.3.(3)]. 

Onr hnal identities of this section display the decomposition of the prodnct of a 
rectangnlarly shaped and a “nearly” rectangnlarly shaped (general linear, symplectic, 
respectively special orthogonal) character. As will become apparent from the proofs, 
these identities follow rather easily from decomposition formnlas for the prodnct of 
a rectangnlarly shaped symplectic, respectively special orthogonal, character and an 
arbitrarily shaped symplectic, respectively special orthogonal, character, which we 
obtain in Proposition 1 in Section 6 from Littelmann’s decomposition formnla [24]. 

Theorem 3. Let n, c, d,p be nonnegative integers. 

First let n > p. If c < d then there holds 

■ sp2n{iid+lT,d^-P);x^^) = ^ sp2^(z/; x±^ ), 

{{d-c)^)CuC{{c+d+l)^) 
oddrows(!//((d—c)”)]) =p 

(3.13) 

and if c > d, 

■ SP2n{{d'^~^,{d-lT)]^^^) = SP2^(z/; X±^), 

((c—d)”)CiyC((c+d)’^) 
oddrows^!//((c—d)”)^ =p 

(3.14) 

Next, if c, d are nonnegative integers or half-integers and p an integer with p < 2d, 
then there hold 

S02n ((c’");X=^^) ■ S02n ((d''“\d-p);X=^^) = ^ S02n(«^;X=^^), 

(|c—d|"“l,c—d)Ci/C((c-|-d)”) 
oddcols^((c-|-d)")/!3]) =p 


(3.15) 
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with the understanding that v ranges over n-orthogonal partitions ifc + d is an integer 
and over n-orthogonal half-partitions if c-\- d is a half-integer, and 

S02n ((c”“\ -c);x=^^) ■ S02n - p) ; 

= ^ S02n((i^l, • ■ • , -«^n);x=^^), (3.16) 

(|c—d)Ci/C((c+d)”) 
evencols^ ((c+d)") / ifj =p 

with the same understanding. 

For c, d nonnegative integers or half-integers and for n > p there holds 
S02n+1 ((C^);X=^^) ■ S02n+1 {{d'^~^, {d - 1)P);X=^^) 

= Cn,p(z^) ■ S02n+l(«^;x=^^), (3.17) 

^|c—d|"“P,(max{c—d,d—c—1})P^ (^(c+d)"^ 

where, with miiy) denoting the multiplicity of I in u, 

Cn,piy) = number of vertical strips of length p — md-c-i{^) 

that can be added to v to obtain another (half-)partition, avoiding the 
{d — c)-th, the (c — (i + l)-st, and the {c -\- d -\- l)-st column of v. 

(3.18) 


Again, the sum in (3.17) is understood to range over partitions if c -\- d is an integer 
and over half-partitions if c-\- d is a half-integer. 

Finally, for c, d nonnegative integers or half-integers and for d> p there holds 

S02n+1 ((c’");x=^^) ■ S02n+1 (((/’"“^ - p); X=^^) 

= <in,p(z/) ■ S02n+i(«^;x=^^), (3.19) 

^|c—d|"“l,max{c—d,d—c—p}^ (^(c+d)"^ 


where dn,p{n) is the number of all horizontal strips a, that can be added to v to obtain 
another (half-)partition, avoiding the {c + d + l)-st column and the n-th row of n, and 
which satisfy the following inequalities: If ai denotes the number of cells of the strip 
a in the i-th row, i — 1, 2,..., n — 1, and if |a| denotes the total number of cells in a, 
then 


\\a\ - c +d - p\ < IZn, IZn + p - IZn-l < \a\ < p 
and 2ai + ■ ■ ■ + 2ai-i + > c — d -h 2p for i — 1,2,... ,n — 1. 


(3.20) 


Also here, the sum in (3.19) is understood to range over partitions if c-\-d is an integer 
and over half-partitions if c-\- d is a half-integer. 

Remark. All these formulas, that is (3.13)-(3.20) and (6.1)-(6.5), have obvious in¬ 
terpretations as decomposition formulas for the tensor product of two representation 



10 


C. KRATTENTHALER 


modules of Sp(2n, C), SO(2n, C), or SO(2n + 1, C). For the interested reader we add 
that, without much additional effort, it is also possible to derive decomposition formu¬ 
las for sp 2 n ■ sp2n d-p); aud S02n ■ S02n (d- 

which are in the spirit of (3.19), respectively (3.17), by slightly modifying 
the arguments in the proofs of (3.19) and (3.17). 

Formulas (3.13)-(3.20) generalize the symplectic and special orthogonal decom¬ 
positions of Okada [27, Theorem 2.5], who proved the p = 0 special cases. Okada 
also proves a decomposition formula for the product of two rectangularly shaped 
general linear characters (Schur functions), thus generalizing Stanley’s results [36, 
Lemma 3.3]. More generally, Carini [2, sec. 3.3, p. 105ff] derived decomposition for¬ 
mulas for the product of two “nearly” rectangularly shaped general linear characters. 

4. Proof of Theorem 1. Here we use decomposition rules of Littlewood [25] 
(see Section A7 of the Appendix). All the notions that appear in this section, 
like Littlewood-Richardson filling (LR-filling), Littlewood-Richardson condition (LR- 
condition), content, etc., are also explained in the Appendix, mainly in Section A6. 

Proof of (3.1). Implicitly, this was already proved in [28, Proof of Lemma 4, Claim 
on p. 558]. However, since there is no explicit statement in [28] and since one would 
have to translate things appropriately, we include a detailed proof of (3.1) here. 

According to (A. 13) we have 


s((c’^ \c-p);x) = ^sp(z/;x) ^ LRJ);"^ 

ly fjs' even 


Hence we have to show that 


/j,, even 


LR 


(c^-fc-p) 


p,,u 


1 oddcols ((c*^)///) = p 
0 otherwise. 


(4.1) 


To see this, suppose that // is a fixed partition whose Ferrers diagram has only even 
columns. Consider a LR-filling of shape ,c — p)/p.. By the LR-condition, there 
is no choice for the entries in the first r — 1 rows. I.e., all the rows except for the r-th 
row are uniquely determined in the way that is exemplified in Figure 1. There is only 
freedom in the r-th row. 
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Figure 1 


It should be observed that the content of the uniquely determined part of the LR- 
filling equals p {c—pr-i, c—pr- 2 , ■ ■ •, c—pi). In particular, this implies that all the 
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columns in (c ^)/fi have odd length, except for columns c — Hr-i +1, c — Hr-i + 2 ,..., c 
(which would have length r) in case that r is even. 

Suppose that n is the content of the complete LR-hlling. Then the LR-condition is 
equivalent to saying that z///i is a horizontal strip (see Appendix Al for the dehnition 
of a horizontal strip). Besides, this horizontal strip is of length c — p — Hr, the length 
of the r-th row of the LR-hlling. Hence, there are c — {c — p — Hr) = p + Hr odd 
columns in {c^^jv if c is odd, and there are c — Hr-i — (c — p — fJ'r) = P + Pr — Pr-i odd 
columns in (c^)/n if r is even. The latter is due to the fact that 1 cannot be an entry 
in the r-th row of the LR-hlling if r is even, hence the horizontal strip has to avoid 
the hrst row. Actually, the number of odd columns is p in both cases, since in case r 
odd we must have Hr = 0 because the Ferrers diagram of n has only even columns, 
and in case r even we have Pr-i = Pr for the same reason. Thus we have shown that, 
given a hxed p whose Ferrers diagram has only even columns, then LRjf ^ q 

only if n is a partition such that the number of odd columns in (c’')/n equals p. In 
particular, this establishes the “otherwise” part of (4.1). 

Conversely, given a partition n C (c^) such that the number of odd columns in 
(c”)/z/ equals p, we claim that there is exactly one partition fi whose Ferrers diagram 
has only even columns and such that LRjf t-p) ^ q And, more precisely, we 
have LRj^^j^ ^^-v) _ which means that there is exactly one LR-hlling of shape 
{c^~^,c — p)/p and content u. Altogether, this would establish (4.1). 

The claim is established by going through the preceding paragraphs, backwards. 
Suppose that n C (c’') is a partition with LR*^^,^ ^ 0, for some p whose Ferrers 

diagram has only even columns. Then fl, dehned as above as {c—pr-i, c—pr- 2 , ..., c— 
Pi), is a partition in which (1) all the columns have length = r mod 2, and (2) which 
is contained in u and differs from n by a horizontal strip of length c — p — pr- It is 
straight-forward to see that (1) and (2) determine p uniquely. Hence, pi, p 2 , ..., Pr-i 
are uniquely determined, thus also pr, since the Ferrers diagram of p has to contain 
only even columns. To be precise, the latter condition implies that pr equals Pr-i if 
r is even, and 0 if r is odd. 

Let p be this uniquely determined partition. To show that LRj^^j^ = 1 it 

remains to see that there is exactly one LR-hlling of shape (c^~^, c—p)/p with content 
u. In fact, as we already observed, the entries of the hrst r — 1 rows of a LR-hlling of 
shape (c’'“^,c — p)/p are uniquely determined. The content of this partial hlling of 
these hrst r — 1 rows is p. Since n, which should be the content of the complete LR- 
hlling, dihers from /i by a horizontal strip, also the entries of the r-th row are uniquely 
determined. To be precise, the length of the i-th row of z///i gives the multiplicity of 
i in the r-th row of the LR-hlling. 

Altogether, this establishes (4.1), and hence (3.1), as desired. □ 

Proof of (3.2). According to (A.14) we have 


s((c^ P, (c-l)P);x) =^o(z/;x) ^ 




,(c-iF) 


/X, fi even 
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Hence we have to show that 

V = I ^ oddrows((c")/^^) =P 

^ \ 0 otherwise. 

fi, fi even ^ 

We could prove this again directly, in a similar style as in the proof of (4.1). However, 
once (4.1) is already known, the companion (4.2) follows straight-forwardly from the 
well-known identity (see [8]) LR)) = LR))/□ 

For the proofs of (3.3) and (3.4) we utilize the following auxiliary result. 

Lemma. Let N be a positive integer. Then, for any partition A = (Ai, A 2 ,..., Aat) 
we have 

'SAr((Ai, A 2 ,..., Ajv); = SAr((Ai — Ajv, Ai — Ajv-i, • • •, Ai — A 2 ,0); x^^). (4.3) 

Proof. This identity follows upon little manipulation from the dual Jacobi-Trudi iden¬ 
tity (the Nagelsbach-Kostka identity, see [26, I, (3.5)]) for Schur functions, 

s(A;x)= det (eA'-i+j(x)), (4.4) 

where em(x) = denotes the elementary symmetric function 

of degree m. For, by (4.4), and because of 

and a similar identity for em{xi, ..., Xn, x~^, 1 ), we have (in the following calcula¬ 
tion em,(x^^) is short for em{xi, xf ^,..., Xn, xf^), respectively for em{xi, xf ^,..., Xn, 
x~^,1), depending on whether N is even or odd) 

Siv((Ai,A 2 ,...,Aiv);x=^^) = det (eA'-i+j(x=^^)) 

= ^ det (eAr_A'+i-i(x±^)) 

= ^^((Ai — Aa/-, Ai — Aiv-i, ■ • •, Ai — A 2 , 0); x^^). 

Proof of (3.3). In (3.1) we specialize x to {xi,xf ^,..., Xn, x~^, 0, 0,...) if V = 2n, 
and to {xi,xf ^,..., Xn, x~^, 1, 0, 0,...) if V = 2n -|- 1, which leads to 

SAr((c’'“\c-p);x^^) = ^ sp^(z/;x^^). (4.5) 

iyC{c^) 

oddcols ^(c’') / ifj =p 
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This gives (3.3) for r < \N/2\ immediately. In case r > [A^/2], on the right-hand 
side of (4.5) the partition z/ conld have more than \N/2\ parts, in which case one 
wonld have to apply modihcation rnles for the corresponding symplectic characters 
spjv(^;^) (see e.g. [10; 39, Theorem 5.4]). However, we circnmvent this difhcnlty by 
appealing to the Lemma above. In fact, by (4.3) and (4.5) we have for r > [A^/2], 

Siv((c’'“^c-p);x^^) = S7v((c^“’',p);x±^) 

oddcols (^(c^“’'+i) / =c—p 


which is (3.3) for r > [A^/2]. □ 

Proof o/(3.4). In (3.1) we set N = 2n -\- 1, p = 0, and snbstitnte ,... 

x~^ ^ 1, 0, 0,...) for X to obtain 

S2n+l((c^);x^^) = Y ^1^2n+l(p;x^^). (4.6) 

oddcols(^(c’’)/p[) =0 

Now let hrst be r < n. The odd symplectic characters on the right-hand side 
of (4.6) are known to decompose in terms of even symplectic characters as (see [30, 
Cor. 8.1; 32, Lemma 9.1 with z = 1 and b = 0]) 

SP2n+l{P',^^^) = Y «I^2n(^;X^^). (4.7) 

pjv a horizontal strip 


Combining this with (4.6) yields (3.4) for r < n, as for any snbpartition v of the 
rectangle {c^) there is exactly one way of adding a horizontal strip to u snch that a 
partition p with oddcols((c’')/p) = 0 is obtained. 

The case r > n of (3.4) then follows from the r < n case by nse of (4.3) with 
N = 2n + 1. □ 

Proof of (3.5). Here, in (3.2) we specialize x to {xi, xf^, ..., x^ 0, 0,...) if = 
2n, and to (xi, ..., Xn, 1, 0, 0,...) if = 2n -|- 1. □ 

5. Proof of Theorem 2. In this section we combine the tableanx description of 
symplectic characters dne to DeConcini and Procesi [3, 4] (see Section A3 of the 
Appendix) and tableanx descriptions of special orthogonal characters dne to Laksh- 
mibai, Mnsili and Seshadri [22, 23, 21] (see Sections A4 and A5 of the Appendix) 
with Robinson-Schensted-Knnth type algorithms. We remark that while there are 
restriction rnles that wonld apply here (they involve ordinary Littlewood-Richardson 
coefficients, see [12, (4.20)-(4.22); 16, Theorem 2.1 with /c = n. Theorem Al]), these 
do not appear to be very helpfnl for onr pnrposes. This is becanse they involve mod¬ 
ihcation rnles for characters, these canse alternating signs, and these in tnrn canse a 
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lot of cancellations, and all this is simply not tractable for the applications that we 
have in mind. 

Proof of (3.6). By (A.4) we know that the left-hand side of (3.6) eqnals 

( 5 - 1 ) 

S a (2n)-symplectic tableau 
of shape (c^-^,(c-l)^) 

On the other hand, by (A. 2 ) we have that the right-hand side of (3.6) eqnals 

{xiX2---Xny- ^ X^. (5.2) 

T an n-tableau 
of shape i/C((2c)^) 
with oddrows(^')=p 


Comparing (5.1) and (5.2), we see that (3.6) will be proved if we can hnd a bijection, 
$ say, between (2n)-symplectic tableanx S of shape (c”“^, (c — 1)^) and n-tableanx 
T with at most 2c colnmns and exactly p odd rows snch that 

= {xiX2 ■ ■ ■ Xn^ ■ : ifT = $(5'). (5.3) 


The bijection that we are going to constrnct proceeds in two steps. In the hrst step 
we map ( 2 n)-symplectic tableanx of shape (c — 1 )^) to certain pairs (see the 

paragraph inclnding (5.4)) by analyzing how these symplectic tableanx look like. In 
the second step we map these pairs to the above described n-tableanx by a Robinson- 
Schensted-Knnth type correspondence. 


First step. Let A be a (2n)-symplectic tableanx of shape (c"'“P, (c — 1)^). By 
the dehnition of (2n)-symplectic tableanx in Section A3 of the Appendix, S' is a (2n)- 
tableanof shape ((2c)"^“P, (2c — 2 )^) snch that colnmns 2 c— 1 , 2 c, colnmns 2 c—3, 2 c— 2 , 
etc., form (2n)-symplectic admissible pairs. An example with n = 6 , c = 4, p = 2 is 
displayed in Fignre 2. 


2 c 
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12 
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9 

11 

11 



7 

10 

10 

11 

12 

12 




Figure 2 


p 


We claim that (2n)-symplectic tableanx S of shape (c — 1)^) are in bijec¬ 

tion with pairs (T, {ci, 62 ,..., e^}), by a bijection $1 say, where T is an n-tablean 
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whose shape has only even rows and is contained in (( 2 c)"^ ( 2 c — 2 )^), and where 

{ci, 62 ,, epj is a set of nnmbers satisfying 


1 < 61 < 62 < ■ ■ ■ < ep < n, 
and 6; ^ [i(m),j(m)] for 1 < / < p, 1 < m < s, 


(6.4) 


given that 


^(1) J(l) 


i(s) j(s) 

are the (2c — l)-st and (2c)-th colnmn of T, snch that 

(a:ia :2 ■ ■■Xn)~'' ■ Xe^ ■ ■ -Xe^ ■ if (T, {ci, 62 ,..., Cp}) = $i(5'). (5.5) 


The constrnction of the bijection $1 is based on an analysis of the symplectic 
tableanx nnder consideration. By Observation 2 in Section A3 of the Appendix, both 
colnmns in a (2n)-symplectic admissible pair have the same nnmber of entries < n. 
Hence, the entries < n in S' form an n-tablean, T say, with only even rows. Of conrse, 
the shape of T is contained in ((2c)(2c — 2)^). In Fignre 2 we have marked the 
area that is covered by entries < n by a bold line. The resnlting tablean is displayed 
in the left half of Fignre 3. 

The next observation is that, given T, we can recover S almost completely, only the 
(2c—l)-st and the (2c)-th colnmn cannot be necessarily recovered. For, all the colnmns 
of S except colnmns 2c—1 and 2c have length n. Thns, by Observation 3 in Section A3 
of the Appendix, if the entries < n in a colnmn of length n are {i(l), i(2),..., i(s)} then 
the entries > n are {n + 1 , n + 2, ..., 2n}\{2n + l —i(l), 2n + l —i(2), ..., 2n-\-l—i{s)}. 
Only for recovering colnmns 2c — 1 and 2c of S from T we need more information 
than jnst the (2c — l)-st and (2c)-th colnmn of T. 

Let the (2c — l)-st and (2c)-th colnmn of S be 


*(1) J(l) 

i{s) j{s) 

i(s + l) j{s + l) 

i{n — p) j{n — p) 


entries < n 


entries > n 


(8.6) 


Observe that by dehnition of a (2n)-symplectic admissible pair (see Dehnition 1 in 
the Appendix) we have 


{i(l),... ,i(s), 2n + 1 - i(s + 1),..., 2n + 1 - i(n - p)} 

= • • • 0'(s), 2n + 1 - i(s + 1),..., 2n + 1 - i(n - p)}. (5.7) 
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Let {ei, 62 ,, Cp} be the complement of this set in {1,2,..., n}. With other words, 
61 , 62 ,..., Cp are the numbers e between 1 and n with the property that neither e nor 
its “conjugate” 2n + 1 — 6 occur in the (2c — l)-st or ( 2 c)-th column of S. Without 
loss of generality we may assume 6 i < 62 < ■ ■ ■ < 6 p. In our running example (recall 
p = 2 ) we have (ci, 62 } = ( 2 , 6 }. 

Obviously, because of (5.7) and the dehnition of T and (ci, 62 ,..., 6 p}, all the 
information about the (2c— l)-st and the (2c)-th column (displayed in (5.6)) is con¬ 
tained in the (2c — l)-st and (2c)-th column of T (the entries < n in (5.6)) and 
(ci, 62 ,..., Cp}. Moreover, because of the dehnition of a (2n)-symplectic admissible 
pair, we have ei ^ [f(m), j(m)], for all I and m. And conversely, if we have 

^( 1 ) J(l) 

■ ■ , {ci, 62 ,...,Cp} 

i{s) j(s) 

such that (5.4) is satished, then (5.6) with 

{t(s + 1),... ,i(n - p)} := {n + 1..., 2n}\{2n -t- 1 - i(l),. ..,2n + l - i(s), 

2n + 1 — ei2n + 1 — Cpj 

{j(s + l),...,j(n-p)} := {n + 1..., 2n}\{2n + 1 - j(l ),..., 2 n + 1 - j(s), 

2n + 1 — ei2n + 1 — Cpj 


will be a (2n)-symplectic admissible pair. 

Hence, we have dehned the desired bijection $ 1 . It is easy to check that the weight 
property (5.5) holds under this correspondence. 

Our running example in Figure 2 is mapped under $1 to the pair in Figure 3. 
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\ 


{ 2 , 6 } 




Figure 3 


Second step. In the second step we construct a bijection $2 between pairs (T, (ci, 
62 ,..., Cp}) satisfying the above conditions including (5.4), and n-tableaux T with at 
most 2 c columns and exactly p odd rows, such that 

yF ^ Xe^-■-Xe^. (5.8) 

Let (T, (ci, 62 ,..., Cp}) be such a pair. We insert Cp, Cp-i,..., ci, in this order, 
into T, according to the following procedure. Let Tq := T. Suppose that, by inserting 
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Cp, Cp-i,..., ep_;_|_i we already formed Ti. Next we insert Cp-i into Ti in the following 
way. Choose the hrst row (from top to bottom) of 7) snch that Cp-i is less than the 
entry in the (2c — l)-st col umn in that row of T). If there is no snch row choose the 
hrst row that does not have an entry in the (2c — l)-st colnmn of that row. Then, 
starting with that row of T), ROW-INSERT Cp-i into T), i.e., (cf. [14, p. 712; 19, 
pp. 87-88]) hnd the leftmost entry in that row that is larger than Cp-p bnmp it and 
replace it by Cp-p if there is none then place Cp-i at the end of that row. If an entry 
was bnmped then repeat this same procednre with the bnmped entry and the next 
row, etc. Thns one obtains Ti+i. Finally, set T = $ 2 ((T, {ci, 62 ,..., Cp})) := Tp. 
Onr rnnning example from Fignre 3 is mapped nnder $2 to the tablean in Fignre 4. 
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Figure 4 


Since T was an n-tablean with only even rows, and since later “insertion paths” 
are (weakly) to the left of previons ones, T is an n-tablean with exactly p odd rows. 
Trivially, (5.8) is satished. 

To show that $2 is a bijection, we have to constrnct the inverse mapping. Take an 
n-tablean T with at most 2c colnmns and exactly p odd rows. Choose the last row 
(from top to bottom) of T that has odd length. Now, starting with that row, perform 
a, slightly modihed, ROW-DELETE (cf. [14, p. 713; 19, pp. 88 ]). Namely, remove the 
last entry, x say, from that row and hnd the rightmost entry, y say, in the previons 
row that is less than x, replace y hj x and repeat this same procednre with y and 
the row before the row that contained y, etc., nntil no row is left to be considered or 
nntil an entry in the (2c)-th col umn wonld be replaced. In the latter case (this is the 
modihcation), do not replace the entry in the ( 2 c)-th colnmn, bnt stop the procednre. 
Thns we obtain an n-tablean Ti with p — 1 odd rows and an entry, ci say, that was 
replaced in the last step of the procednre. This procednre is repeated with Ti, thns 
obtaining T 2 and 62 , etc. In the end we obtain Tp, which is an n-tablean with only 
even rows, and in the conrse of onr algorithm we obtained the elements ci, 62 ,..., Cp. 
By standard properties of ROW-INSERT and ROW-DELETE (cf. [34]), it is not 
difhcnlt to see that this algorithm exactly reverses $ 2 , step by step. In particnlar, 
the start of ROW-INSERT in a row that is possibly different from the first and the 
modified ending of ROW-DELETE complement each other exactly. 

The composition $2 o $1 is by definition the desired bijection between (2n)-sym- 
plectic tableanx S of shape (c"^“P, (c — 1)^) and n-tableanx T with at most 2c colnmns 
and exactly p odd rows. From (5.5) and (5.8) the weight property (5.3) follows 
immediately. This completes the proof of (3.6). □ 

Proof of (3.7). Using (A. 6 ) and (A. 2 ) in (3.7), we see that (3.7) will be proved if we 
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can find a bijection, if' say, between (2n)-orthogonal tableanx S of shape c — p) 

and n-tableanx T with at most 2c colnmns and exactly p colnmns with parity different 
from n (by which we mean that the lengths of the colnmns have parity different from 
n) snch that 

= {xiX2 ■ ■ ■ Xn)~''■ : ifT = \l/(5'). (5.9) 

Again, the bijection that we are going to constrnct proceeds in two steps. In the 
first step we map ( 2 n)-orthogonal tableanx of shape (c"^“^,c —p) to certain pairs 
(see the paragraph inclnding (5.10)) by analyzing how these orthogonal tableanx look 
like. In the second step we map these pairs to the above described n-tableanx by 
Robinson-Schensted-Knnth insertion. 


First step. Let A be a (2n)-orthogonal tableanx of shape (c”“^, c — p). By Obser¬ 
vation 1 in Section A5 of the Appendix, S consists of a pair (As, S 2 ) of ( 2 n)-tableanx, 
As being of shape (( 2 c —p)”) and each colnmn of which containing an even nnmber of 
entries > n, S 2 being of shape (p"^), each colnmn of which containing an odd nnmber 
of entries > n, and all the entries in the first row of S 2 being at most n, snch that the 
concatenation U is a (2n)-tablean, where is the tablean arising from S 2 by 
replacing the topmost element, say, in colnmn i of S 2 by its “conjngate” 2 n -|-1 — e^, 
for alH = 1 , 2 ,... ,p, and by rearranging the colnmns in increasing order. An example 
with n = 6 , c = 7/2, p = 2 is displayed in the left half of Fignre 5. The right half 
shows the concatenation S^, U S 2 (note that ci = 1, 62 = 2). 
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We claim that (2n)-orthogonal tableanx S of shape (c"^“^, c—p) are in bijection with 
pairs (T, {ci, 62 ,..., Cp}), by a bijection Ti say, where T is an n-tablean whose shape 
has only colnmns with the same parity as n and is contained in (( 2 c)"^“^, 2 c — p), and 
where {ci, 62 ,..., Cp} is a set of nnmbers satisfying 


1 < Cl < 62 < ■ ■ ■ < Cp < n, 

Ci is less than the topmost element of the (2c — p -\- f)-th colnmn of T, 
snch that 


(5.10) 


(X*')® = (1112 '"In) ° ■ le, ■ ■ -le, ■ , if (r, {C], 62, ■ ■ ■ , Cp}) = '11 (S) . (5.11) 
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The construction of the bijection Ti is based on an analysis of the orthogonal 
tableaux under consideration. By dehnition, all the columns in Ss U S 2 contain an 
even number of entries > n. Hence, the entries < n in S's U form an n-tableau, 
T say, with all columns having the same parity as n. Of course, the shape of T is 
contained in ((2c)"^“^, 2c — p). In the right half of Figure 5 we have marked the area 
that is covered by entries < n by a bold line. The resulting tableau is displayed in 
the left half of Figure 6 . 

Now, let \I/i(5') be dehned by (T, {ci, 62 ,..., Cp}), where, as before, is the top¬ 
most element of the i-th column of 82 - Our running example in Figure 5 is mapped 
under Ti to the pair in Figure 6 . It is obvious that (5.10) and (5.11) hold under this 
mapping. Besides, it is trivial to recover S from (T, {ci, 62 ,..., Cp}). Hence, Ti is a 
bijection, as desired. 
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Figure 6 


Second step. In the second step we construct a bijection \l /2 between pairs (T, {ci, 
62 ,..., Cp}) satisfying the above conditions including (5.10), and n-tableaux T with 
at most 2 c columns and exactly p columns of parity different from n, such that 


X 


T 




• X 


(5.12) 


Let (T, {ei, 62 ,..., Cp}) be such a pair. We ROW-INSERT ei, 62 ,..., Cp, in this 
order, into T. (The reader should observe the differences to the algorithm in the 
second step of the proof of (3.6): Here we use genuine ROW-INSERT, without any 
modihcation, as opposed to the proof of (3.6). Furthermore, the order in which 
the elements ci, 62 ,..., Cp are inserted is exactly the reversed order in the proof of 
(3.6).) I.e., (cf. [14, p. 712; 19, pp. 87-88]) let Tq := T. Suppose that, by inserting 
Cl, 62 ,..., 6 / we already formed Ti. Next we insert 6 ;+i into Ti. Namely, we hud the 
leftmost entry in the hrst row of T) that is larger than 6 ;-|-i, bump it and replace it by 
6 ;+i, if there is none we place 6 /_(_i at the end of that row. If an entry was bumped then 
we repeat this same procedure with the bumped entry and the next row, etc. Thus 
one obtains the tableau T,+i. Finally, we set T = T 2 ((T, {ci, 62 ,..., 6 p})) := Tp. 
Our running example from Figure 6 is mapped under 4/2 to the tableau in Figure 7. 
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Figure 7 


Since T was an n-tablean with all colnmns having the same parity as n, and since 
later “insertion paths” are strictly to the right of previons ones, T is an n-tablean 
with exactly p colnmns of parity different from n. Moreover, since is less than the 
topmost element of the (2c — p + t)-th colnmn of T, the insertion process will at no 
stage prodnce more than 2c colnmns. Hence, T has at most 2c colnmns. Trivially, 
(5.12) is satished. 

To show that \l /2 is a bijection, we have to constrnct the inverse mapping. Take 
an n-tablean T with at most 2c colnmns and exactly p colnmns of parity different 
from n. Choose the rightmost colnmn of T that has parity different from n. Snppose 
that the bottommost entry in this col umn is in row I. Now, starting with that row, 
perform ROW-DELETE, i.e., (cf. [14, p. 713; 19, pp. 88]) remove the last entry, xj 
say, from the /-th row and hnd the rightmost entry, xj-i say, in the (/ — l)-st row 
that is less than xj, replace xj-i by xj and repeat this same procednre with xj-i and 
the (/ — 2)-nd row, etc., nntil no row is left to be considered. Thns we obtain an n- 
tablean Ti with p — 1 colnmns of parity different from n and an entry, Cp say, that was 
replaced in the last step of the procednre. This procednre is repeated with Ti, thns 
obtaining T 2 and Cp-i, etc. In the end we obtain Tp, which is an n-tablean with all 
colnmns having the same parity as n, and in the conrse of onr algorithm we obtained 
the elements Cp, Cp-i,..., ci. Again, by standard properties of ROW-INSERT and 
ROW-DELETE (cf. [34]), it is not difficnlt to see that this algorithm exactly reverses 
4 / 2 , step by step. 

The composition 4 / 204/1 is by dehnition the desired bijection between (2n)-orth- 
ogonal tableanx S of shape (c"^“^, c — p) and n-tableanx T with at most 2c colnmns 
and exactly p colnmns of parity different from n. From (5.11) and (5.12) the weight 
property (5.9) follows immediately. This completes the proof of (3.7). □ 

Proof of (3.8). Here we have to deal with (2n -|- l)-orthogonal tableanx. Since these 
are not too far from (2n)-symplectic tableanx, the argnments here are very similar 
to those in the proof of (3.6). In fact, the basic steps are the same, only the details 
differ. So we shall be sometimes sketchy and provide details only if necessary. 

Using (A.5) and (A.2) in (3.8), we see that (3.8) will be proved if we can hnd a 
bijection, 0 say, between (2n-|-1)-orthogonal tableanx S of shape (c”“^, (c —1)^) and 
pairs (T, a), where T is an n-tablean whose shape is contained in {{2c)'^), and where 
a is a vertical strip of length p on the rim of u that avoids the (2c)-th colnmn, snch 
that 


(x±i)S _ (^ 1^2 ... a;^) ° if (T, cr) = 0(5'), for some a. 


(5.13) 
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Again, we proceed in two steps. 


First step. Let A be a (2n + l)-orthogonal tableanx of shape ( 0 ""“^, (c — 1)^). By 
the dehnition of (2n + l)-orthogonal tableanx in Section A4 of the Appendix, S is 
a (2n)-tablean of shape ((2c)"^“^, (2c — 2)^) snch that colnmns 2c — 1, 2c, colnmns 
2c — 3, 2c — 2, etc., form (2n + l)-orthogonal admissible pairs. An example with 
n = 6, c = 4, p = 3 is displayed in Fignre 8. 
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Figure 8 


p 


We claim that (2n + l)-orthogonal tableanx S of shape (c — 1)^) are in 

bijection with pairs (T, {ci, 62 ,..., Cp}), by a bijection 0i say, where T is an n- 
tablean contained in (2c — 2 )^), and where {ci, 62 ,..., Cp} is a set of nnmbers 

satisfying 

1 < Cl < 62 < ■ ■ ■ < Cp < n, 

ei ^ [t(m), j(m)] for 1 < / < p, 1 < m < s, (5-14) 

and ei ^ [t(s + 1 ), n] for 1 < / < p, 

given that 

*( 1 ) J(l) 

t(s) j{s) 

i{s + 1) 

i{t) 

are the ( 2 c — l)-st and ( 2 c)-th colu mn of T, snch that 

= {xiX2 ■ ■ Xe, ■ ■ -Tep ■ , if (T, { 61 , 62 , . . . , 6 p}) = 0l(S'). (5.15) 

The constrnction of the bijection 0i is based on an analysis of the orthogonal 
tableanx nnder consideration. Clearly, the entries < n in S' form an n-tablean, T 
say, whose shape is contained in ((2c)"^“^, (2c — 2)^). Note that in difference to the 
symplectic case, here T is not necessarily a tablean with only even rows. In Fignre 8 
we have marked the area that is covered by entries < n by a bold line. The resnlting 
tablean is displayed in the left half of Fignre 9. 
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Now, let the (2c — l)-st and (2c)-th column of S be 


n > 


n < 


t(l) 

m 



i{t) 

j(s + 1 ) 

i{t + 1 ) 


i{n — p) 

j{n-p) 


< n 


> n 


(5.16) 


As in the symplectic case, we dehne {ci, 62 ,..., Cp} to be the set of numbers e between 
1 and n with the property that neither e nor its “conjugate” 2 n + 1 — e occur in the 
(2c — l)-st or ( 2 c)-th column of S. Again, without loss of generality we may assume 
Cl < 62 < ■ ■ ■ < Cp. In our running example (recall p = 3) we have { 61 , 62 , 63 } = 
(2, 3, 6}. From the dehnition of a (2n + l)-orthogonal admissible pair (see Dehnition 2 
in the Appendix) it follows that the numbers 61 , 62 ,..., 6 p satisfy (5.14). Note that 
(5.14) differs from the “symplectic analogue” (5.4) by the additional condition in the 
third line. 

We dehne 0i(5') to be (T, (ci, 62 ,..., 6 p}). Our running example in Figure 8 is 
mapped under 0i to the pair in Figure 9. 
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Figure 9 
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{2,3,6} 




As in the symplectic case, it can be shown that S can be uniquely recovered from 
(T, { 61 , 62 ,..., 6 p}). Also, it is easy to check that the weight property (5.15) holds 
under this correspondence. 

Second step. In the second step we construct a bijection 02 between pairs (T, {ci, 
62 ,..., 6 p}) satisfying (5.14) as before, and pairs (T, a), where T is an n-tableau whose 
shape is contained in (( 2 c)"^), and where cr is a vertical strip of length p on the rim of 
u that avoids the ( 2 c)-th column, such that 

■ ■ -Xep ■ (5.17) 

To obtain T from such a pair (T, {ci, 62 ,..., 6 p}) we use the mapping $2 from the 
second step of the proof of (3.6). On the other hand, a is dehned to be the vertical 
strip by which the shapes of T and T differ. Thus, our running example from Figure 9 
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is mapped under 02 to the pair in Figure 10. There, the vertical strip a is visualized 
by bold lines embedded in the shape of T. 



\ 


} 


By dehnition of 02 (recall that an element is never inserted into the (2c)-th column 
of some Ti), a is a vertical strip on the rim of the shape of T that avoids the (2c)-th 
column. It is obvious that (5.17) holds. 


The composition 02o0i is by dehnition the desired bijection between (2n + l)-orth- 
ogonal tableaux S of shape (c”“^, (c — 1)^) and pairs (T, cr), where T is an n-tableau 
whose shape is contained in ((2c)”), and where a is a vertical strip of length p on the 
rim of u that avoids the (2c)-th column. From (5.15) and (5.17) the weight property 
(5.13) follows immediately. This completes the proof of (3.8). □ 


Proof of (3.10). Here we proceed very similarly to the preceding proof of (3.8). 

Using (A.5) and (A.2) in (3.10), we see that (3.10) will be proved if we can hnd a 
bijection, O say, between (2n + l)-orthogonal tableaux S of shape (c”“^, c — p) and 
pairs (T, a), where T is an n-tableau whose shape is contained in ((2c)”), and where 
a is a horizontal strip of length p on the rim of u such that the f-th cell of the strip 
comes before the (2c — 2p + 2f)-th column, such that 


— {xiX 2 ■ ■ ■ Xn) if (T, a) = 0(5”), for some cr. (5.18) 


Again, we proceed in two steps. 


First step. Let S' be a (2n -|- l)-orthogonal tableau of shape (c”“^, c — p). By the 
dehnition of (2n -|- l)-orthogonal tableaux in Section A4 of the Appendix, S is a (2n)- 
tableau of shape ((2c)”“^, 2c — 2p) such that columns 2c—1, 2c, columns 2 c—3, 2 c—2 , 
etc., form (2n -|- l)-orthogonal admissible pairs. An example with n = 6, c = 7/2, 
p = 3 is displayed in Figure 11. 
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Figure 11 


We claim that (2n + l)-orthogonal tableaux S of shape ^,c — p) are in bijection 

with pairs (T, (ei, 62 ,..., Cp)), by a bijection 0i say, where T is an n-tableau contained 
in (( 2 c)"^“^, 2 c — 2 p), and where (ci, 62 ,, Cp) is a vector of numbers (we dehnitely 
mean vector here, i.e., the order of the numbers is important) satisfying 


1 ^ Cl, 02, . . . , Cp ^ 71, 

ei ^ [ii{m),ji{m)] for 1 < / < p, 1 <m < sp (5.19) 

ei ^ [ii{si + 1), n] for 1 < / < p, 


and for all m, 

\{1,2,... ,m}\{ei,ji{l),... ,ji{si)}\ < I {1,2 ,... ,m}\{ei+i,ii+i{l),... 

(5.20) 

given that 


*/(l) 

31 O-) 

iiisi) 


ii{si + 1) 


kiti) 



are the (2c — 2p + 2/ — l)-st and (2c — 2p + 2/)-th column of T, such that 
(x^^^)*^ == {X1X2 - ■ -Xn^ ■ ■ ■ ■ Xe^ ■ x'^, if (T, (ci, 62 , . . . , Cp}) = Oi(5'). (5.21) 

Again, the construction of the bijection Oi is based on an analysis of the orthogonal 
tableaux under consideration. Clearly, the entries < n in A form an n-tableau, T say, 
whose shape is contained in ((2c)"^“^, 2c — 2p). In Figure 11 we have marked the area 
that is covered by entries < n by a bold line. The resulting tableau is displayed in 
the left half of Figure 12. 

If we apply the paragraph containing (5.16) with p = 1, then we see that, with 
ei being the number that together with its conjugate 2 n -|- 1 — does not appear in 
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the (2c — 2p + 2l — l)-st or (2c — 2p + 2/)-th column of S', / = 1, 2,... ,p, the map Oi 
dehned by 

S (T, (ei,e2,...,ep)) 

dehnes the desired bijection. In particular, the fact that also the entries > n in S 
form a (skew) tableau is reflected by condition (5.20). For, the entries > n from the 
(2c — 2p + 2/)-th column of S are 


{n + l,n + 2,... ,2n}\{2n + 1 - cp 2n + 1 - jz(l),..., 2n + 1 -ji{si)}, 


and the entries > n from the (2c — 2p + 21 + l)-st column of S are 


{n + l,n + 2,... ,2n}\{2n + 1 - ei+i,2n + 1 - b+i(l),..., 2n + 1 - b+i(b+i)}- 


That all entries > n from the (2c — 2p + 2/)-th column of S are less or equal than 
their right neighbours from the (2c — 2p + 2l + l)-st column of S, is exactly equivalent 
to requiring 


{2n, 2n - 1,..., 2n - m}\{2n + 1 - cp 2n + 1 - ..., 2n + 1 - jz(sz)}| 

< I {2n, 2n — 1,..., 2n — m} 

\{2n + 1 - ei+i,2n + 1 - b+i(l),..., 2n + 1 - b+i(fz+i)} 


for all m, which is clearly equivalent to (5.20). 

Again, it is easy to check that the weight property (5.21) holds under this corre¬ 
spondence. Our running example in Figure 11 is mapped under Oi to the pair in 
Figure 12. 
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Figure 12 


Second step. In the second step we construct a bijection Q 2 between pairs (T, (ci, 
62 ,..., Cp)) satisfying (5.19) as before, and pairs (T, a), where T is an n-tableau whose 
shape is contained in (( 2 c)"^), and where a is a is a horizontal strip of length p on the 
rim of u such that the t-th cell of the strip comes before the ( 2 c — 2p + 2 t)-th column, 
such that 

• -Xep ■ (5.22) 

Let (T, (ei, 62 ,..., 6 p)) be such a pair. We insert 61 , 62 ,..., 6 p, in this order, into 
T, according to a procedure that is very similar to $ 2 , or 02 , which were used in 
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the proofs of (3.6), respectively (3.8). That we have to modify these procedures 
is due to the fact that ei satishes a condition, namely (5.19), that depends on the 
( 2 c — 2 p + 2 / — l)-st and ( 2 c — 2p + 2 /)-th column of T, and not just on the ( 2 c — l)-st 
and (2c)-th column as was the case in (5.4) or (5.14). Let Tq := T. Suppose that, 
by inserting ei,e 2 ,...,ei we already formed Ti. Next we insert ei+i into Ti in the 
following way. Choose the hrst row (from top to bottom) of T such that ei+i is less 
than the entry in the (2c — 2p + 2/ — l)-st column in that row of Ti. If there is no 
such row choose the hrst row that does not have an entry in the ( 2 c — 2 p + 2 / — l)-st 
column of that row. Then, starting with that row of T, ROW-INSERT eij^i into T, 
see the dehnition of $2 in the second step of the proof of (3.6). Thus one obtains the 
tableau Finally, set T = Q 2 {{T, (ci, 62 ,..., Cp))) := Tp. On the other hand, a 

is dehned to be the vertical strip (it is indeed a vertical strip, as will be shown in a 
moment) by which the shapes of T and T differ. Thus, our running example from 
Figure 12 is mapped under O 2 to the pair in Figure 13. There, the vertical strip a is 
visualized by bold lines embedded in the shape of T. 



It is obvious that (5.22) holds under this correspondence. Moreover, it is immediate 
from the dehnition of a as a result of the above insertion procedure, that the f-th cell 
of a comes before the (2c — 2p + 2f)-th column. However, it is not so immediate that 
a is indeed a vertical strip. This will be established next. We will be done if we are 
able to show that later “insertion paths” are strictly to the right of previous ones. It 
suffices to consider two successive insertion paths. 


Let the columns 2c — 2p -|- 2/ — 1, 2c — 2p -|- 2/, 2c — 2p -|- 2/ -|- 1, 2c — 2p - 1 - 2/ -|- 2 be 
given by the four columns in Figure 14 (ignore for the moment ‘c; —and —;►’) 
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*/(l) 

3iiX) 

U+i(l) 




ei+i k+iiv) 

jl+liv) 

ii{u) 

ji{u) 

k+i{si+i) 

ji+i{si+i) 

iiisi) 
k{si + 1) 


k+i{ti+i) 


k{ti) 


Figure 14 



Suppose that the insertion of e; would start in row u, as is symbolized by ei —> ii{u) 
in Figure 14, and that the subsequent insertion of eij^i would start in row n, as is 
symbolized by ei —> ii^i{v) in Figure 14. This would mean that we have 

ji{u - 1) < ei < ii{u) (5.23) 

and 

ji+i{v - 1) < ei+i < ii+i{v). (5.24) 

During insertion of ei+i, hrst the element ei+i bumps ii^i{v) (which is an element 
of the (2c — 2p + 21 + l)-st column of T) or an element to the left of ii^i{v) in the 
same row. Let us further suppose that until row w — 1, w > v, elements of the 
(2c — 2p + 21 + l)-st column are bumped, i.e., ii^i{v) bumps + 1), ... , hnally 

— 2) bumps — 1). This would mean 


ji{v + 1) < ii+i{v),. ..,ji{w - 1) < ii+iiw - 2). (5.25) 

We suppose that then in row w the insertion path jumps to the left of the (2c — 
2p + 21 + l)-st column (note that the case w — v covers the case that ei+i bumps an 
element to the left of ii^i{v)), which means that 

ii+i{w - 1) < ji{w), or ei+i < ji{v) in case w = V. (5.26) 

We do not care what happens afterwards. 

We claim that u < w and that ei < ii+i{w — 1), respectively ei < cz+i in case 
w = V. This would imply that in the n-th row the insertion path caused by eij^i 
is strictly to the right of the insertion path caused by ei and therefore has to stay 
strictly to the right from thereon, by an elementary property of ROW-INSERT. And 
this is what we want to show. 
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For proving the claim we consider (5.20) with m = — 1), respectively m = 

e;-|-i in case w — v. For this choice of m, the right-hand side of (5.20) eqnals ii+i{w — 
1) — rc since we have 

q+i(l) < ii+i{2) < ■ ■ ■ < ii+i{w - 1), 

and by (5.24), 

ei+i < ii+i{v) < ■ ■ ■ < ii+i{w - 1), 
respectively eqnals ei+i — n in case w — v since by (5.24) we have 

b+i(l) < ■ ■ • < ii+i{v - 1) < ji+i{v - 1) < ez+i < b+i(^^)- 

Hence, the left-hand side of (5.20) is bonnded above by — 1) — rc, respectively 

ez-i-i — n in case w = v. 

On the other hand, the left-hand side of (5.20) is at least ii+i{w—l)—w, respectively 
ei+i — n in case w — v, and eqnal to the lower bonnd only if ei < ii+i{w — 1), 
respectively ei < ei+i in case w = v. For, by (5.25) and (5.26) we have 

ji{l) < ■ ■ ■ < ji{w - 1) < ii+i{w - 2) < ii+i{w - 1) < ji{w), 

and in case w = v we have by (5.24) and (5.26) 

ji{l) < ■ ■ ■ < jiiv - 1) < ji+i{v - 1) < ei+i < ji{v). 

So the lower bonnd can only be reached if ei < ii+i{w — 1), respectively ei < ei+i in 
case w = V, which is what we wanted to show. 

Snmmarizing, we have shown that indeed e; < ii+i{w — 1), respectively ei < e;_|_i 
in case w = v. Combining this with (5.23) and (5.26), we obtain the ineqnality chain 

ji{u -1) < ei< ii+i{w - 1) (respectively ei+i) < ji{w), 

hence ji{u — l) < ji{w). Since colnmns are strictly increasing, this immediately implies 
u < w, as desired. 

To show that 4/2 is a bijection, we have to constrnct the inverse mapping. Experi¬ 
enced with three other similar proofs, this is rather straight-forward. Let (T, a) be a 
pair, where T is an n-tablean whose shape is contained in ((2c)"^), and where a is a 
horizontal strip of length p on the rim of u snch that the f-th cell of the strip comes 
before the (2c — 2p + 2f)-th colnmn. For I = 1, 2,... ,p start a ROW-DELETE with 
the entry of T that is located in the cell corresponding to the l-th cell of a (connted 
from the right), bnt stop before an entry in the (2c — 21 + 2)-th colnmn wonld be 
bnmped. This procednre reverses the algorithm O 2 , step by step. 

The composition Q 2 ° bli is by dehnition the desired bijection between (2n -|- 1)- 
orthogonal tableanx S of shape (c”“^, c — p) and pairs (T, cr), where T is an n-tablean 
whose shape is contained in (( 20 )"^), and where a is a horizontal strip of length p on 
the rim of u snch that the f-th cell of the strip comes before the (2c — 2i + 2)-nd 
colnmn. From (5.21) and (5.22) the weight property (5.18) follows immediately. This 
completes the proof of (3.10). □ 
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6. Proof of Theorem 3. In this section we nse Littelmann’s extension [24] of the 
Littlewood-Richardson rnle to symplectic and special orthogonal characters. This ex¬ 
tension is described in Section A6 of the Appendix. We remark that while there are 
other rnles for the decomposition of the prodnct of two symplectic or orthogonal char¬ 
acters involving ordinary Littlewood-Richardson coefficients (the Newell-Littlewood 
rules, see [13, Theorem 4.1; 15, Cor. 2.5.3/Prop. 2.5.2; 39, Theorem 5.3]), these do not 
appear to be very helpfnl for onr pnrposes. Again, this is becanse they involve modi¬ 
fication rnles for characters, these canse alternating signs, and these in tnrn canse a 
lot of cancellations, and all this is simply not tractable for the applications that we 
have in mind. 

Before we move on to the proofs of (3.13)-(3.20) itself, in Proposition 1 we snpply 
decomposition formnlas for the prodnct of a rectangnlarly shaped symplectic, respec¬ 
tively special orthogonal, character and an arbitrarily shaped character of the same 
type. All these expansions involve slightly modified Littlewood-Richardson coefh- 
cients, which even rednce to ordinary Littlewood-Richardson coefhcients in a nnmber 
of cases, see the Remark after Proposition 1. Bnt there are no alternating signs here, 
and hence there is no cancellation. The formnlas (3.13)-(3.20) then follow rather 
easily from (6.1)-(6.5). 

Proposition 1. For any nonnegative integer c and any partition A with at most n 
parts there holds 

zyCA+(c’^) ^tC((2c)’^) 

/i. even 


(‘H even’ means that all the rows of n are even), where \c) is the number of 

LR-fillings F of shape {u + (c"'))/A with content fi and with the additional property 
that 


if there is an entry e in the n-th row of F, in column j say (see Section Al 

in the Appendix how columns are counted), then F must contain at least (6.2) 

2c — 2j -|- 1 other entries e to the right of column j. 

Next, for any nonnegative integer or half-integer c and any partition or half-parti¬ 
tion A with at most n parts there holds 

S02n+1 ((c");x^^) ■ S02n+1 (A;x^^) = ^ S 02 n +1 («^; ^ \c), 

/xC((2c)”) 

(6,3) 

with the understanding that v ranges over partitions if X-\- (c"") is a partition and over 

half-partitions z/A-|-(c"') is a half-partition, and where { c ) is defined as before. 

(Again, see Section Al in the Appendix how columns are counted; in particular, in 
condition (6.2) the column index j ranges over the integers if X is a partition and 
over half-integers if X is a half-partition.) 
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Finally, for any nonnegative integer or half-integer c and any {2n)-orthogonal par¬ 
tition or half-partition A with at most n parts there holds 


S02n ((c’^);x=^^) ■ S02n (A;X=^^) 


LR 


!^CA+(c") /iC((2c)") 

oddcols(((2c)”)//j,^ =0 


A,/I 


(c) , (6.4) 


'+Fn, 


where LRa,/x (c) is the number of LR-fillings F of shape {n + (c"'))/A with content 
fi and with the additional property that for £ = 1,2,... ,2c holds: 


If the subfilling that arises from F by deleting the rightmost 2c —£ entries 
1, the rightmost 2c — £ entries 2, ... , the rightmost 2c — £ entries n (if 
there are less than 2c — £ entries of some size then delete all of these) 
has shape u{£)/X then h'{£)n-i + i'{£)n > £. 


(6.5) 


Again, the sum in (O.f) is understood to range over n-orthogonal partitions z/A + (c"') 
is an n-orthogonal partition and over n-orthogonal half-partitions if X -\- (c"') is an 
n-orthogonal half-partition. 

Remark. It should be noted that in case X^ > c the condition (6.2) is void, so that 
the coefficients ^(c) which appear in (6.1) and (6.3) reduce to the ordinary 

Littlewood-Richardson coefficients \ Likewise, if A^-i + An > 2c the condi- 


‘'A,/x 


+R"), 


tion (6.5) is void, so that the coefficients LR;^^^ (c) which appear in (6.4) reduce to 

the ordinary Littlewood-Richardson coefficients LR^^^^ ^ 

Proof of Proposition 1. For convenience, we start with the proof of (6.3). 

Proof of (6.3). By (A. 10) with Xn( ■) = ■S 02 n+i( ■; x^^) we know that 


S02n+1 ((c’");x=^^) ■ S02n+1 ^ S 02 n+i (A + con(T); x=^^), 

T 


( 6 . 6 ) 


where the sum is over all (2n + l)-orthogonal tableaux T of shape {c^) such that for 
all £ = 1, 2,..., 2c the vector u{£) := A + con(T(£)) is in the Weyl chamber (A.8) of 
type B, i.e., satisfies 

iy{£)i>iy{£)2>--->iy{£)n>0. (6.7) 

The content con(T) of T is defined after (A.3). 

By comparing (6.6) with (6.3) we see that (6.3) will be proved once we construct 
a bijection, T say, between (2n + l)-orthogonal tableaux T of shape (c"^) and content 
p which satisfy (6.7) for £ = 1, 2,..., 2c and LR-fillings F of shape (A -|- p -|- (c”))/A 
which satisfy property (6.2). 

The bijection T is defined as follows. Let A be fixed and let T be a (2n -|- 1)- 
orthogonal tableau of shape (c"^) with content p. By Observation 2 in Section A4 of 
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the Appendix, (2n + l)-orthogonal tableaux of shape (c"^) are nothing else but (2n)- 
tableaux of shape ((2c)"^) where each column contains one of e or 2n + 1 — e for all 
e = 1, 2,..., n. An example with n = 5 and c = 5/2 is displayed in the left half of 
Figure 15. It satishes the required property that A + con(T(£)) is in the Weyl chamber 
of type S, £ = 1, 2,..., 2c for A = (4,4, 3,1,1), as is easily checked. 





1 

0 





2 




2 

3 



1 

2 

3 




T 

T 

4 




1 

1 

1 

4 

6 

2 

3 

4 

6 

7 

T 

4 

5 

8 

8 

5 

5 

8 

9 

9 

7 

9 

9 

10 

10 


Figure 15 

To obtain the image of T under T, we construct a sequence Fq, Fi,..., F 2 c of 
Filings by reading T column-wise, from right to left. The desired filling F will then 
be defined to be the last filling, F 2 c- Define Fq to be the only filling of the shape A/A 
(which is of course the empty Filing). Suppose that we already constructed F^. To 
obtain F^_|_i, we add for f = 1, 2,..., n an entry e to row i of F^ if i is an entry in the 
(2c — £)-th column and the e-th row of T. As already announced, we deFne F to be 
F 2 c- Thus, with A = (4, 4, 3,1,1), our tableau in the left half of Figure 15 is mapped 
by T to the Filing in the right half of Figure 15. 

It is straight-forward from this construction that the mapping T can be reversed, 
step by step. So we shall be done if we show that, given that T is mapped to F by 
T, T is a (2n-t-l)-orthogonal tableau of shape (c"^) with content p if and only if F is 
a LR-Flling of shape (A -|- p -f (c"^))/A satisfying (6.2). We provide the details only 
for the forward implication. Since the arguments for the backward implication are 
similar, the reader will have no difFculties to Fll in the respective details. 

Let T be a (2n-t-l)-orthogonal tableau of shape (c"^) with content p that is mapped 
by T to the Filing F. What we have to show is that F is an n-tableau, i.e., that 
entries are weakly increasing along rows and strictly increasing along columns, that 
the LR-condition holds, that the shape of F is (A -|- con(T) -|- (c’^))/A, and that F 
satishes (6.2). The reader is advised to keep the example of Figure 15 in mind. It 
will help to follow the subsequent arguments. 

For the Frst statement, let e and / be entries in the Fth row of F, e being the 
left neighbour of /. Then, by construction of T, e was caused by some entry i in the 
e-th row and Cg-th column, say, of T, while / was caused by some entry i in the f-th 
row and c/-th column, say, of T. Since / is to the right of e, / was added “later”, 
hence c/ < Cg. Now, since T is a tableau, the entry i in column Cf cannot be higher 
than the entry i in column Cg. Therefore we have e < /. This holds for any left-right 
neighbours in F, so rows are weakly increasing, as desired. To see that columns of F 
are strictly increasing, we consider entries e and / in the same column of F, e being 
the top neighbour of /. Let e be located in the Fth row of F (and so / be located in 
the {i -|- l)-st row of F). Then, by construction of T, e was caused by some entry i in 
the e-th row and Cg-th column, say, of T, while / was caused by some entry (t + 1) in 
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the /-th row and c/-th column, say, of T. It is easily seen by induction that for all i the 
shape of the partial hlling Fi is given by (A + con(T(£)) + ((£/2)"'))/A. In particular, 
because of (6.7), this implies that the “outer shape” of is always “well-behaved” 
in the sense that lower rows always terminate earlier than higher rows. Therefore the 
entry f of F was added “later” then the entry e, which means that the column c/ of 
T must be weakly to the left of column Cg. We already know that there is an entry i 
in column Cg and an entry (t + 1) in column c/. Since column c/ is located weakly to 
the left of column Cg, the entry {i -|- 1) must be in a lower row than the entry i. As 
the entry (t + 1) is located in the /-th row and the entry i is located in the e-th row, 
this means nothing else than e < /. This holds for any top-bottom neighbours in F, 
so columns are strictly increasing, as desired. 

Now we turn to the LR-condition. We have to show that, while reading the entries 
of F row-wise from top to bottom, and in each row from right to left, at any stage 
we have 

number of I’s > number of 2’s > number of 3’s > ■ ■ ■ . 

Now, by construction of T, each entry in F corresponds to some entry in T. Thus, 
to the above described reading of the entries of F there corresponds the following 
reading of the entries of T: First read the entries 1 in T, from left to right, then the 
entries 2, from left to right, then the entries 3, etc. The LR-condition is equivalent to 
saying that at any stage during this reading of T the number of entries read from the 
hrst row is greater or equal the number of entries read from the second row, which in 
turn is greater or equal the number of entries read from the third row, etc. But this 
is obviously true because T is a tableau. 

What regards the shape, we already noticed that for all i the shape of Fi is given 
by (A -I- con(T(£)) -|- ((£/2)”))/A. Hence F = F 2 c has shape (A -|- con(T) -|- (c"'))/A. 

Finally, we want to show that F satishes (6.2). Let e be a hxed entry in the n-th 
row and j-th column of F. Write again z/(£) = A -|- con(T(£)). By assumption, z/(£) 
lies in the Weyl chamber of type B, see (6.7). In particular, we have iy{i)n > 0. Now, 
we already observed that 

(A + con(T(£)) + {{i/2r))/X = {u{e) + ((£/2)-))/A 

is the shape of Fg. Hence, the condition > 0 is equivalent to saying that Fg 

contains an entry in the n-th row that is located in the (£/2)-th, respectively (£-|-l)/2- 
th, column (depending on whether A is a partition or half-partition). (In passing, we 
note that the i — 2c case of this fact implies that F = F 2 c contains an entry in the 
c-th, respectively (c-|-l/2)-th, column. Hence the shape of F can indeed be written in 
the form (n -|- (c"^))/A with u a partition or half-partition.) On the other hand, Fg is 
the subhlling of F that arises by deleting the rightmost 2c —i entries 1, the rightmost 
2c — i entries 2, ... , 2c — £ entries e, etc., from F. Now, suppose that the hxed entry 
e in the n-th row and j’-th column is also contained in Fg, i.e., j < [i + l)/2. Then 
there must be at least 2c — i entries e in F to the right of the j’-th column. This last 
property holds for all i with j < {i + l)/2. Hence there must be at least 2c — {2j — 1) 
entries e in F to the right of the j’-th column. This is exactly what we wanted to 
show. 
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Thus, the proof of (6.3) is complete. 

Proof of (6.1). By (A.10) with Xn{ ■) = ■sp 2 n( • 5 x=*=^) we know that 

S 1 ^ 2 n(-^ + COn(T); X±^), (6.8) 

T 

where the sum is over all (2n)-symplectic tableaux T of shape (c^) such that for all 
£ = 1,2,..., 2c the vector u{£) := A + con(T(£)) is in the Weyl chamber (A.8) of 
type (7, i.e., satishes (6.7). (Recall that the Weyl chambers of types B and C are 
the same.) By comparing (6.8) with (6.1) we see that (6.1) will be proved once we 
construct a bijection between (2n)-symplectic tableaux T of shape (c^) with content 
p and LR-hllings F of shape (A + p+(c"^))/A with even content which satisfy property 
( 6 . 2 ), 

As bijection we can take the mapping T from the preceding proof of (6.3). We 
only have to observe (see Observation 3 in Section A3 of the Appendix) that (2n)- 
symplectic tableaux T of shape (c^) are the same as (2n + l)-orthogonal tableaux of 
shape {c^) with the additional property that the entries < n form a subtableau with 
only even rows. Suppose that T is mapped by T to F. Then the length of the i-th row 
of this subtableau of T is the same as the number of occurences of i in the hlling F. 
In other words, the shape of the subtableau equals the content of the corresponding 
hlling F. Since the shape of the subtableau is even, the hlling must have even content, 
as desired. The hnal observation is that since (6.7) holds here, too, the hlling must 
again satisfy (6.2). 

Proof of (6.4). By (A. 10) with Xn{-) = S 02 n{ ■ ;x=^^) we know that 

S02n ((c'");x=^^) ■ S02n (A;x=^^) = ^s02n(A + con(T);x=^^), (6.9) 

T 

where the sum is over all (2n)-orthogonal tableaux T of shape (c”) such that for all 
£ = 1, 2,..., 2c the vector u{£) := A + con(T(£)) is in the Weyl chamber (A.9) of type 
D. By comparing (6.9) with (6.4) we see that (6.4) will be proved once we construct 
a bijection between (2n)-orthogonal tableaux T of shape (c”) with content p and LR- 
hllings F of shape (A -|- p -1- (c”))/A and content p where all the columns of p have 
the same parity as n and where property (6.5) is satished. 

Again, we can take the mapping T from the proof of (6.3) as the bijection. Here, 
this is because of the observation (see Observation 1 in Section A5 of the Appendix, 
with A„_i = A„ = c) that (2n)-orthogonal tableaux T of shape (c"^) are the same as 
(2n-|-l)-orthogonal tableaux of shape (c"^) with the additional property that the entries 
< n form a subtableau whose shape has only columns of the same parity as n. Again, 
since the shape of the subtableau equals the content of the corresponding hlling, 
the content p of the hlling must have columns of the same parity as n throughout, as 
desired. This time we have to impose (6.5) (instead of (6.2)) since z/(£) = A-|-con(T(£)) 
has to be in the Weyl chamber of type D (and not of type B or C). 

This completes the proof of the Proposition. □ 
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Now we are in the position to prove (3.13)~(3.20). 

Proof of (3.13). We apply (6.1) with X = {{d + Becanse of the assnmp- 

tion c < d, the Remark after Proposition 1 applies, which says that the coefficients 
\c) rednce to ordinary Littlewood-Richardson coefficients \ Then, 

obvionsly, (3.13) is eqnivalent to the claim 

{ 1 {{d - c)^) C u C {{c + d + 1)^) 

and oddrows(z//((d-c)^)) =p (6.10) 

0 otherwise. 

^ even 

Let F be a LR-hlling of shape (z/+(c”))/(((i+l)^, dP~P) with even content. Becanse 
of the LR-condition, almost all the entries of F are nniqnely determined. To be precise, 
except for the entries in colnmn d + 1, all the entries in the t-th row have to eqnal i 
thronghont, t = 1, 2,..., n, as is exemplihed in Fignre 16. 



Figure 16 


However, also the entries in the (d + l)-st colnmn of F are nniqnely determined 
because the content of F should be even. Namely, i occurs in the (d+ l)-st column if 
and only if the number of the other entries i, which is z/j + c — (d + 1), is odd. In our 
example in Figure 16, the entries in the (d + l)-st col um n would have to be 1,2,5. 
Now, suppose that F contains exactly q entries in the (d + l)-st column, q < n — p 
of course. Equivalently, u — (z/i,..., i^p+q, d — c,..., d — c). Then there are exactly 
q quantities + c — (d + 1), i < p + q, that are odd, plus n — p — q quantities 
z/i + c — (d + 1) = —1 for t = p + g + 1, p + g + 2,..., n. Therefore, the number of 
odd rows in z//((d + 1 — c)'^) is exactly q + {n — p — q) = n — p^ or equivalently, the 
number of odd rows in z//((d — c)'^) is exactly p. 

Summarizing, we have shown that the left-hand side in (6.10) is different from zero 
only if ((d — c)"') C u C {{d + c + I)"") and oddrows(z//((d — c)"')) = p, the inclusions 
being trivial constraints. In addition, we have also seen that there is exactly one 
LR-hlling under those conditions. Thus, (6.10), and thus also (3.13), is established. 
□ 


Proof of ( 3 . 14 ). Here we apply (6.1) with A = p, (d — 1)^). Now the assumption 
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is c> d, so the Remark after Proposition 1 does not apply. We have to show 

{ 1 ((c-d)”) C z/C ((c + d)^) 

and oddrows(z//((c-d)”)) =p (6.11) 
0 otherwise. 

fj. even 

From (6.10) it follows directly that the left-hand side in (6.11) can only be non-zero 
if {{d — c — !)"■) C z/ C ((c-|- (i)"") and if oddrows(z //{{d — 1 — c)"')) = n — p. Note that 
the latter condition is eqnivalent to oddrows(z//((c — d)"")) = p (if this makes sense, 
i.e., if ((c — d)^) C u). It also follows from (6.10) that if the left-hand hand side of 
(6.11) is non-zero then it can only be 1. So, what remains to see is that it is non-zero 
if and only if in addition {{c — d)^) C u, or eqnivalently, c — d < Un. 

We begin with the forward implication. Snppose that there is a LR-hlling F of 
shape (z/ -|- (c"^))/((i"'“P, (d — 1)^) with even content satisfying (6.2). If c = d there 
is nothing to show. So let c > d. Then, by considering the shape of F, we see that 
there mnst be an entry in the n-th row and (d -|- l)-st colnmn of F. (If there is no 
entry in the (d -|- l)-st colnmn of F, then + c < d. So, z/^ < d — c < 0, which 
is impossible since u has to be a partition.) Clearly, this entry eqnals n since it is 
located in the last row of a colnmn of length n. Now, condition (6.2) applied to this 
entry says that there mnst be 2c — 2 (d -1- 1) -|- 1 = 2c — 2d — 1 more entries n to the 
right of this colnmn. All of them necessarily have to be in the n-th row of F, hence 
+ c > (d -|- 1) -|- (2c — 2d — 1), or eqnivalently, z/^ > c — d, as desired. 

For the backward implication, assnme z/^ > c—d. We have to establish the existence 
of a LR-hlling of shape (z/ -|- (c"'))/(d"^“^, (d— 1)^) with even content satisfying (6.2). 
This LR-hlling can only be the nniqnely determined hlling that was described in the 
proof of (3.13). In fact, the argnments of the hrst paragraph of this proof and those 
in the proof of (3.13) actnally show that this nniqnely determined LR-hlling satishes 
all the reqnired properties, except for possibly (6.2) for the entry in the d-th colnmn. 

We now verify (6.2) for this entry, e say, by distingnishing between too cases. First 
let e 7 ^ n. Condition (6.2) wonld reqnire that there are 2c — 2d -|- 1 more entries e 
to the right. All these entries necessarily have to be located in the e-th row. Hence, 
condition (6.2) wonld reqnire Ue + c — d > 2c — 2d -|- 1. Now, becanse of Un > c — d, we 
have Ue + c — d > Un + c — d > 2c —2d. So the total nnmber of e’s, which is z/g + c —d-f-l, 
is at least 2c — 2d -|- 1. Since the content of the hlling is even, this nnmber mnst be 
even. So it is actnally at least 2c — 2d -|- 2. Hence, z/g + c — d -|- 1 > 2c — 2d -|- 2, or 
eqnivalently, z/g-t-c — d>2c — 2d-|-l, as reqnired. 

Now let e — n. Condition (6.2) wonld reqnire that there are 2c —2d-|-l more entries 
n to the right. Now, the total nnmber of n’s (all of them are located in the n-th row) 
eqnals Un + c — d+l. Becanse of Un > c — d this nnmber is at least 2c — 2d -|- 1. Again, 
since the content of the hlling is even, this nnmber mnst be even. So it is actnally at 
least 2c — 2d -|- 2. Hence, there are at least 2c — 2d -|- 1 more n’s to the right of the n 
in the d-th colnmn, as reqnired. 

This completes the proof of (3.14). □ 
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Proof of (3.15). We apply (6.4) with A 


{dP h — p). Thus, we have to show that 


mC((2c)") 


LR 


V+{cP 

,d—p) 




(c) 


oddcols(^((2c)”)//x^ =0 


1 (|c — C — h) C Z/ C ((c + (i)"') 

and oddcols(((c + d)"")///) =p 
0 otherwise. 


( 6 . 12 ) 

Let F be a LR-hlling of shape (z/ + {c^))/{d‘^ ^,d — p) with content p, where 
oddcols(((2c)'^)/p) = 0. For later use we note right here that the left-hand side of 
(6.12) can only be non-zero if 


iy<p{{c+dr). 


(6.13) 


Similarly here, because of the LR-condition, the entries in the hrst n — 1 rows are 
uniquely determined. To be precise, all the entries in the i-th row have to equal i, 
i — 1,2,... ,n — 1, as is exemplihed in Figure 17. 


d 



Q 


d 



Q 


Figure 17 


However, also the entries in the n-th row of F are uniquely determined because the 
content p of F should satisfy oddcols(((2c)"^)///) = 0. For convenience, let F denote 
the subhlling of F consisting of the hrst n — 1 rows of F. Namely, there have to be 
as many entries i in the n-th row of F as there are columns in F of length i — 1 with 
parity different from n, i = 1,2,... ,n. (On the side we note that therefore all the 
entries in the n-th row have the same parity as n.) In our examples in Figure 17, the 
entries in the n-th row would have to be 2, 2,4, 6 and 2, 2, 4, 6, 6, 6, respectively. Now, 
suppose that F contains exactly q entries in the n-th row. Then there are exactly q 
columns in F with parity different from n. If q < p then the number of columns of F 
whose parity is different from n equals the aforementioned q columns plus the p — q 
empty columns d — p + q + l,d — p + q + 2,...,d, see the left Filing in Figure 17. If 
q > p then the number of columns of F whose parity is different from n equals the 
aforementioned q columns minus the q — p columns (i-|-l,(i-|-2, ...,(i-|-( 3 ' — pof length 
n, see the right Filing in Figure 17. Hence, in both cases the number of columns of F 
whose parity is diFerent from n equals p, or equivalently, oddcols(((c -|- d)^)/u'j = p. 
(Recall that by (6.13) the shape u is indeed contained in ((c -|- d)”).) 

Summarizing, we have shown that the left-hand side of (6.12) is diFerent from 
zero only if z/ C ((c -|- d)^) and oddcols(((c -|- (i)"')/z/) = p. We have also shown 
that if the left-hand side is non-zero, it can only be 1 since there is at most one 
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LR-filling. So, what remains is to show that it is non-zero if and only if in addition 
(|c — c — d) C z/. 

Again, we begin with the forward implication. Suppose that there is a LR-hlling F 
of shape (z/-t-(c"'))/((i"'“^, d—p) with oddcols(((2c)"')///) = 0, where p is the content of 
F, satisfying (6.5). Because of ,d—p) C z/-|-(c"'), we have ((d—d—c—p) C 
u. Hence, we will be done if we can show c — d < 

We already noted that the hrst n — 1 rows of F are uniquely determined, in partic¬ 
ular, the (n — l)-st row of F contains only entries n — 1. Now we apply condition (6.5) 
with i = c d — r'n-i- (Note that in (6.5) this choice of i has the effect of removing 
2c — i = I'n-i + c — d entries of each size. In particular, all the entries from (n — l)-st 
row are removed.) Since the content p of F satishes oddcols(((2c)"')///) = 0, the 
number of n’s must equal the number of (n — l)’s in F. Therefore the number of n’s, 
all of which have to be in the n-th row, is at least as large as the number of (n — l)’s 
in the (n — l)-st row of F, the latter being Un-i + c — d. Therefore condition (6.5) 
with i = c + d — r'n-i reads 

(c -F Un-l - [Un-l + C - d)) + [c+ Un - (z^n-1 + C-d)) >C-^d - Z/n-1- 

Simplifying, we obtain > c — d, which is what we wanted. 

For the backward implication, assume > c—d. Recall that there is a uniquely de¬ 
termined LR-hlling of shape (u + (c”))/(d"^“^, d — p) with content p, where 
oddcols(((2c)’^)/p) = 0. What has to be shown is that it also satishes (6.5). A 
careful reading of the previous paragraph reveals that Un > c — d is actually equiva¬ 
lent to condition (6.5) with i = c-\-d — r'n-i- Now, a moment’s thought will convince 
the reader that in our particular situation (the (n — l)-st row consists of (n — l)’s 
throughout, £ = c -\- d — fn-i in (6.5) therefore empties the complete (n — l)-st 
row), condition (6.5) holds for all £ if and only if it holds for the particular choice 
£ = c -\- d — r'n-i- Thus, (6.5) is established. 

This completes the proof of (3.15). □ 

Proof of (3.16). Equation (3.16) is an easy corollary of equation (3.15) withp replaced 
by 2d—p. This is due to the observation that the substitution Xn l/x^i in a character 
■S 02 n(A; x=*=^) has the effect 

'5Z52n ((-^17 • ■ • 7 -^n —1 j ^n): ^ ) 17 . /t- '502n ((-^1 j • • • : ^n — 1: -^n); ^ j 

which follows easily from the dehnition (2.12). □ 

Proof of (3.17). We apply (6.3) with A = (d"^“P, (d—1)^). Note that the only difference 
between (6.3) and (6.1) is that in (6.1) the partition p is required to be even. Hence, 
we can use those arguments from the proofs of (3.13) (here one has to replace d by 
d — 1 and p by n — p) and (3.14) that do not rely on this requirement. 

Again, we have to consider LR-hllings F of shape (u -|- (c”))/(d^“^, (d — 1)^) that 
satisfy (6.2), but with arbitrary content. So, again all entries to the right of the d-th 
column are uniquely determined, and in the same way, see Figure 16. But the entries 
in the d-th column are not unique now. In fact, there is almost complete freedom, the 
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constraints being that the entries along the colnmns have to be strictly increasing, 
i.e., each entry of a particnlar size can only occnr once, that the total content n of F 
has to be a partition of conrse, and that (6.2) mnst be satished. It is easy to see that 
the hrst and second constraint are eqnivalent to {fx + {d'^))/{u + (c"")) being a vertical 
strip of length p — rud-c-iiy) avoiding the d-th colnmn and, becanse of p F ((2c)"^), 
the (d + 2c + l)-st colnmn. Note that the latter “avoidance” conditions give the hrst 
and third “avoidance” condition in (3.18) when colnmns are connted with respect to 
z/, i.e., when everything is shifted back by c. 

The inclnsion (|c — (max{d — c — l,c — d})^) C p follows from the trivial 

inclnsion ((d — {d — c — 1)^) C z/, and from the fact that c — d < z/^ if c > d, 

which is shown in the same way as in the proof of (3.14). 

Finally, we claim that if c > d the vertical strip {p + (d”))/(z/ + (c”)) has to avoid 
the (2c — d + l)-st colnmn. Note that this “avoidance” condition gives the second 
“avoidance” condition in (3.18) when colnmns are connted with respect to z/, i.e., 
when everything is shifted back by c. The former “avoidance” condition comes from 
considering condition (6.2) for the entry in the n-th row and d-th colnmn of the Filing 
F. See the analogons considerations at the end of the proof of (3.14). The difference 
here is that the content p can be arbitrary. Hence the argnment in the proof of (3.14) 
that the nnmber of e’s or n’s is even does not apply here. We leave the details to the 
reader. 

This hnishes the proof of (3.17). □ 

Sketch of proof of (3.19). As in the preceding proof of (3.17) we apply (6.3), this time 
with A = {dF~^,d — p). The argnments are very similar here. We have to consider 
LR-hllings F of shape {y + (c"'))/(d^“^, d — p) with arbitrary content. As in the 
proof of (3.15), all the entries in the hrst n — 1 rows of snch a Filing are nniqnely 
determined, and in the same way, see Fignre 17. Bnt the entries in the n-th row 
are not. They can be chosen arbitrarily as long as the LR-condition and (6.2) are 
satished. The LR-condition implies that, with p denoting the content of the Filing, 
{pi -|- d,..., Pn-i + d)/{ui + c,..., I'n-i + c) is a horizontal strip, a say. Of conrse, 
the length of cr, which eqnals the nnmber of entries < n — 1 in the n-th row of the 
LR-Flling, is at most iZn F c — d -\- p, the length of the n-th row of the Filing. Note 
that this is one part of the hrst ineqnality in (3.20). At the same time, the length of 
a is at most p. For, there cannot be more than p entries < n — 1 in the n-th row since 
the entry in the (d -|- l)-st col umn (which is a colnmn of length n) and the n-th row 
has to be n. Note that this proves the second part of the second ineqnality in (3.20). 
Moreover, the LR-condition applied to entries n — 1 and n implies the Frst part of 
the second ineqnality in (3.20). Finally, condition (6.2) implies the third ineqnality 
in (3.20), and, when applied to the Frst entry n in the n-th row, that the length of a 
is at least c — d + p — Un, the latter being the missing part of the Frst ineqnality in 
(3.20). We leave it to the reader to check this in detail. 

The inclnsion (|c — d|"^“^,max{c — d, d — c — p}) C z/ follows on the one hand 
from the trivial inclnsion {{d — , d — c — p) C z/, and on the other hand from 

c — d < z/„, which is derived from the Frst and second ineqnality in (3.20) as follows: 
l^n F —W\ + c — d + p > c — d. □ 
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7. Applications to plane partitions and tableanx ennmeration. We now 

apply results from Section 3 to derive some enumeration results for plane partitions 
of trapezoidal shape and for tableaux. Recall [28] that the trapezoidal shape (A, N — 
2,..., A — 2r + 2) is an array of cells with r rows, each row indented by one cell to 
the right with respect to the previous row, and N — 2i + 2 cells in row i. A plane 
partition of shape (A, A — 2,..., A — 2r + 2) is a hlling of the trapezoidal shape 
(A, A — 2,..., A — 2r + 2) with nonnegative integers (note that we allow 0 as an 
entry) such that entries along rows and columns are weakly decreasing. 

We begin with an application of (3.3). The second statement in the theorem below, 
(7.2), is a result of Proctor [28, Corollary on p. 554]. 

Theorem 4. The number of plane partitions of trapezoidal shape (A, A — 2,..., 
A — 2r + 2) with entries between 0 and c and where the entries on the main diago¬ 
nal form a partition with exactly p columns of parity different from r (equivalently, 
= p if r is even, respectively = p if r is odd, 

with an denoting the first entry in row i), equals 

fp -\- r — 1 
c\ \ P 
p) /A — r + c 

V P 


n 

l<i<r 

l<j<c 


N — i-\- j 
i+j -I 


(7.1) 


In particular, the number of plane partitions of trapezoidal shape (A, A — 2,..., A — 
2r + 2) with entries between 0 and c equals 


n 

l<i<r 


A + 1 — i + j 
i + j-1 


l<j<c 


(7,2) 


Proof. We use (3.3) with Xi = 1, i = 1,2,..., [A/2J. For this choice of xfs, the Schur 
functions reduce to a number (which is the dimension of the corresponding irreducible 
representation of GL(A, C)) which has a nice closed form (see [6, Ex. A.30.(ii); 26, 
I, Ex. 4 on p. 45; 39, Theorem 4.4]), and is therefore easily computed. Thus, the 
left-hand side of (3.3) turns into (7.1). On the right-hand side of (3.3), we have a 
certain sum of symplectic characters evaluated at Xi = 1, i = 1,2,..., [A/2J. Now, 
beside the (2n)-symplectic tableaux of DeConcini and Procesi in Section A3 of the 
Appendix there are other symplectic tableaux. Namely, the (even) symplectic char¬ 
acters be described by King’s [11] symplectic tableaux of shape 

A (see also [33, Theorem 4.2; 39, Theorem 2.3]), and the (odd) symplectic characters 
sp 2 n+i(-^; can also be described by Proctor’s [30] odd symplectic tableaux of 
shape A (see also [33, Theorem 4.2 with ^ = 1]). There is a uniform dehnition. Let 
A = 2n or A = 2n -|- 1. Then a King/Proctor symplectic tableau of shape A is an 
A-tableau of shape A such that the entries in the i-th row are at least 2i — l for all i. 
Thus, the right-hand side of (3.3) can be interpreted as the number of King/Proctor 
symplectic tableaux with entries < A and of some shape u, where u is contained in 
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(c’’) and oddcols((c’')/z/) = p. These tableaux are now translated into plane par¬ 
titions of trapezoidal shape as described in [31, bottom of p. 295]. Namely, given 
such a King/Proctor symplectic tableau, replace each entry e by 2n -|- 1 — e, then 
interpret each row of the resulting array as a partition and replace it by its conju¬ 
gate partition. Next, shift the f-th row by {i — 1) cells to the right, i = 1,2,..., to 
obtain a plane partition of “shifted” shape that is contained in the trapezoidal shape 
(N, N — 2,N — 2r + 2). Finally, place a zero in each cell of the trapezoidal shape 
that is not yet hlled. It is easy to see that during this transformation the lengths 
of the rows of a King/Proctor symplectic tableau become the entries on the main 
diagonal of the resulting plane partition of trapezoidal shape. This establishes the 
hrst assertion of Theorem 4. 

The number in (7.2) is obtained by summing the numbers in (7.1) over all p, 
by means of the Vandermonde sum (cf. [35, (1.7.7)]). This completes the proof of 
Theorem 4. □ 

The next two theorems give applications of Theorem 2. For the proof of these 
theorems we need a few determinant evaluations that are listed in the Lemma below. 
We remark that the evaluations (7.3), (7.4), (7.5) are basically the Weyl denominator 
factorizations of types C, B, D, respectively (cf. [6, Lemma 24.3, Ex. A.52, Ex. A.62, 
Ex. A.66]). 

Lemma. The following identities hold true: 


det (x^- — X - 
l<i,j<n * * 


n 

{xi---Xn)~'^ JJ {{Xi - Xj){l - XiXj)) JJ(a:--l), (7.3) 

l<i<j<n i=l 


det (x^- 

l< 2 ,_ 7 <n 


i-i/2 


{xi---Xn) {{xi - Xj){l - XiXj)) ]^(a:i - 1), (7.4) 

l<i<j <n 


i=l 


det {xi ^+x^^^ = 2 ■ {xi ■ ■ ■ Xn) 

l<i,j <71 


JJ {{xi- Xj){l- XiXj)), (7.5) 




det (xj+x-^) 

l<i,j<n * * 


{Xl---Xn) {{Xj - Xj){l - XjXj)) y^^ekjxi, . . .,Xn)‘^, 

l<i<j<n fc=0 

(7.6) 


det {xi -t-Xj 

l<i,j<n * * 


= {xi---Xn) JJ {{Xi- Xj){l- XiXj)) JJ(Xi-M). (7.7) 

l<z<j<n 


i=l 
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Proof. Identities (7.3)-(7.5), and (7.7) are readily proved by the standard argnment 
that proves Vandermonde-type determinant evalnations. 

For (7.6) there is a little bit of work to do. First, by reversing the order of colnmns, 
and adding some factors, we rewrite the determinant in (7.6) as 


det ix^- 


n+l-j _j_ 


det [xf ^ X 




det (x^ 


(7.8) 




Next we observe that becanse of (2.12) the qnotient of determinants is one half of 
the orthogonal character 02 n ((I? 1 1); (recall that 02 n(A;x=*=^) is the snm 

of so 2 n(A; x=*^^) and so 2 n(A“; x=*^^) if ^ 0). In addition, we reverse the order of 
colnmns in the single determinant in (7.8) to obtain for (7.8) 


1 


((l,l,...,l);x±‘) det (x’ 

l<z,j <n 


,J-1 


+ Xi 


-0-1)' 


(7.9) 


By the “orthogonal Jacobi-Trndi identity” [6, Cor. 24.45; 15, Theorem 2.3.3, (6)], the 
orthogonal character in (7.9) is nothing else bnt the elementary symmetric f un ction 
en{xi,xf^,... ,XmXf^). Moreover, the determinant in (7.9) can be evalnated by 
(7.5). Thns, the expression (7.9) becomes 


e„(a:i,a:i \ ... ,a:n,T^^) ■ {xi---Xn) {{xi - Xj){l - XiXj)). (7.10) 

l<z<j<n 

The elementary symmetric f un ction in (7.10) can be transformed as follows: 


fxi,x 


1 5 ■ • • 5 ^ n : 


) = '^ekixi,...,Xn) ■en-fc(Ti\...,a:^^) 


k=0 


= {xi---Xn) ^y^^ekjxi, . . . ,Xn)‘^. 
k=0 


Flagging this into (7.10) completes the proof of (7.6). □ 

As a hrst application of Theorem 2 we give new proofs of two theorems of the 
anthor [19, Theorems 21 and 11] by means of (3.6). These are rehnements of the 
Bender-Knnth and MacMahon (ex-)conjectnres, see [19, Sections 3.3, 4.3] for more 
information and references. In order to be able to formnlate the Theorem, we have 
to introdnce a few g-notations. We write [a]q 1 — g", [n]g! := [l](7[2]g ■ ■ ■ [n]^, 
[0]g! := 1, and 




( N? ■[n-l]q---[n-k + l]q 

n 

k 

= < 



Q 

[ 0 


k>0 

k<0 


The base q in [q;]^, [n]^!, and will in most cases be omitted. Only if the base is 
different from q it will be explicitly stated. 
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Theorem 5. Let n(T) denote the sum of all the entries of a tableau T. The gener¬ 
ating function ^ for tableaux T with p odd rows, with at most c columns, and 
with entries between 1 and n is given by 


P’-l 

[2r + p] 


n + 2r 
n 


n -\-2r -\- p 
n 


n 


[2r + i + j] 

[* + j] 


if c — 2r 


(7,11) 


and 




n 


1 <z<j <n 


[2r + i + j] 

[i + j] 


if c = 2r -\-1. 


(7.12) 


The generating function ^ for tableaux T with p odd rows, with at most c 
columns, and with only odd entries which lie between 1 and 2n — 1, is given by 


2 [2r + 2 p]q 2 [r]q 2 

[2r + p]q 2 [r + p]q 2 



n -|- 2r 




n 



r n -|- 2r -|- p 


L 

n 

. 



X 


[r + i]g2 


i=l 




n 

l<i<j<n 


[2r + i + j]q 2 


ifc = 2r (7.13) 


and 



n 


[r + i]q 2 


T i=i 




n 

l<z<j<n 


[2r + i + j]q 2 

V+j]q^ 


if c = 2r -\-1. 


(7.14) 


Proof. For proving (7.11), replace Xi by gb i — 1,2, ...,n, in (3.6). By (2.13), the 
left-hand side of (3.6) can be written as a qnotient of two determinants which for this 
choice of xfs factor by means of (7.3). On the other hand, by (A.2) the right-hand 
side of (3.6) with Xi = , i — 1,2,... ,n, clearly eqnals the generating fnnction for 

the tableanx in the first assertion of Theorem 5. 

The expression (7.12) is obtained withont mnch effort from the p = 0 case of (7.11) 
as is described in [19, Proof of (4.3.2b)]. 

The proof of (7.13) proceeds in the same way as the proof of (7.11). To point ont 
the differences, instead of snbstitnting now snbstitnte for Xi, i = 1,2,... ,n, 

and nse (7.4) instead of (7.3). The expression (7.14) is obtained withont mnch effort 
from the p = 0 case of (7.13) as is described in [19, Proof of (3.3.3b)]. □ 

Onr second application of Theorem 2 leads to different refinements of the Bender- 
Knnth and MacMahon (ex-)conjectnres which are new. What we do is to nse (3.7) to 
obtain a “colnmn-analogne” of Theorem 5. Now, the tableanx generating fnnctions 
in qnestion do not factor completely in terms of cyclotomic polynomials, bnt it is still 
possible to write the resnlts in a reasonably compact form. 
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Theorem 6. Let n(T) denote the sum of all the entries of a tableau T. The gener¬ 
ating function ^ for tableaux T with at most c columns, p of which being odd, 
and with entries between 1 and n, is given by 


p + n — 1 

P 

c — p + n — 1 
c — p 

77 [c + i + j — ‘2] 

2 

c + n — 1 

c 

l<i<j<n ^ 


X 


qHk+c-i) 

k=0 



n — 1 
k 


+ q-P 


n — 1 
k-1 


+ (l_gC-2p) 


i=l 


(7.15) 


The generating function ^ for tableaux T with at most c columns, p of which 
being odd, and with only odd entries which lie between 1 and 2n — 1, is given by 



p + n — 1 


c — p + n — 1 



r,P. 

p 


c — p 

n 

Airr_ 

1 



c + n — 1 



c 



[c + i + j- 2 ]g 2 

-*■ 1 + [i+j—2]„2 

= 1 ^ l<i<j<n I. ' W 


n 


n —1 n—1 

X ((1+n (1++(1 - n 

7=1 7 = 1 


Proof. Since it is simpler, we start with the proof of (7.16). Replace c by cj^ and 
snbstitnte for xi in (3.7), i = 1, 2,..., n. By (2.12), the left-hand side of (3.7) can 
be written as a qnotient of a snm of two determinants by another determinant. The 
determinants, with this choice of XiS, factor by means of (7.4) and (7.7), respectively. 
After simplification we arrive at the expression 


p + n — 1 


c — p + n — 1 

P 


c — p 



c + n — 1 



c 

(j2 




C + i+j -2]q2 


X 


111 + ^2(7-1) 11 L+^-_2] + 

i=l ^ l<i<j<n L ' J \q 

n—1 \ 

(1+ q^-^p) n (1+n 


T7—1 


7 = 1 


7=1 


Note that the only difference from (7.16) is the term (—1)"^ in the last line of (7.17). 

On the other hand, by (A.2) the right-hand side of (3.7) with c replaced by c/2 and 
with Xi = i — 1, 2,..., n, eqnals the generating fnnction for tableanx with at 

most c colnmns, p of which having parity different from n, and with only odd entries 
which lie between 1 and 2n — 1. Now we distingnish between n even or odd. If n is 
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even then the right-hand side of (3.7) with these replacements eqnals the generating 
fnnction for the tableanx in the second assertion of Theorem 6, and the expressions 

(7.17) and (7.16) agree. If n is odd then the right-hand side of (3.7), with the above 
replacements and with p replaced by c — p, eqnals the generating f un ction for the 
tableanx in the second assertion of Theorem 6, and, as a few manipnlations show, the 
expressions (7.17), with p replaced by c — p, and (7.16) agree. 

For the proof of (7.15) we proceed in the same manner, only that things are more 
complicated here. Now we replace c by c/2 and snbstitnte g* for i = 1, 2,..., n, in 
(3.7). Again, by (2.12), the left-hand side of (3.7) can be written as a qnotient of a snm 
of two determinants by another determinant. The determinant in the denominator, 
with this choice of x^’s, can be evalnated by means of (7.5). The second determinant 
in the nnmerator, with this choice of xds, can be evalnated by means of (7.3). The 
hrst determinant in the nnmerator is evalnated by means of (7.6). After simplihcation 
we arrive at the expression 




p -|- n — 1 

c — p + n — 1 

P 

c — p 


c -t- n — 1 
c 


n 

l<z<j<n 


c + i + j - 2] 

[* + /] 


X 


k=0 


k{k+c-l) I ^k 


n — 1 
k 


+ q~ 


n — 1 
k-1 


n—1 

+ (_l)»(l_g-2p) JJ(l-5-+«)|, 

i=l 


(7,18) 


Again, note that the only difference from (7.15) is the term (—1)"^ in the last line of 

(7.18). 

On the other hand, by (A.2) the right-hand side of (3.7) with c replaced by c/2 
and with = g*, * = 2,..., n, eqnals the generating fnnction for tableanx with at 

most c colnmns, p of which having parity different from n, and with entries between 1 
and n. Again, we distingnish between n even or odd. If n is even then the right-hand 
side of (3.7) with these replacements eqnals the generating fnnction for the tableanx 
in the hrst assertion of Theorem 6, and the expressions (7.18) and (7.15) agree. If 
n is odd then the right-hand side of (3.7), with the above replacements and with 
p is replaced by c — p, eqnals the generating fnnction for the tableanx in the hrst 
assertion of Theorem 6, and, as a few manipnlations show, the expressions (7.18), 
with p replaced by c — p, and (7.15) agree. In particnlar, in the last step it is nsed 
that 


k=0 


k{k+c—l) I qk 


n — 1 
k 


+ q 


—c+p 


n — 1 
k-1 


k=0 


k{k+c—l) j ^k 


n — 1 
k 


+ q 


-p 


n — 1 
k-1 


which is verihed by expanding the sqnares. 
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This finishes the proof of Theorem 6. □ 

Since the results in Theorem 6 are completely combinatorial in nature, it would of 
course be desirable to find a combinatorial proof of Theorem 6. The papers [19, 20] 
(see also [18]) contain combinatorial proofs for the assertions in Theorem 5, in par¬ 
ticular avoiding representation theory. But these are already quite difficult. The fact 
that the expressions in (7.15) and (7.16) are more complicated than those in (7.11)- 
(7.14) supports the suspicion that it will be even much harder to find combinatorial 
proofs for the assertions in Theorem 6. 

We conclude by providing an application of Theorem 3, (3.13), (3.14). 

Theorem 7. The number of plane partitions of trapezoidal shape (2n, 2n — 2,..., 
2) with entries between 0 and N where the entries on the main diagonal are at least 
M and exactly p of them have parity different from N is 



fN-M + n\ 

\ n-Fl ) -p-r {N + M+ i+j){N - M+ i + j) 
(N-M + n + p\ {i+jr 

V n + 1 ) 

if N + M is even 


(7,19) 


and 



I'M + M + n + l\ 

V n + 1 ) -|-r {N + M + 1 + i + j){N - M - 1 + i + j) 

fN + M + 1 + n + p\ (f + jT 

V n+1 J 


if N + M is odd. (7.20) 


In particular, the number of plane partitions of trapezoidal shape (2n, 2n — 2,..., 2) 
with entries between 0 and N where the entries on the main diagonal are at least M 
equals 


2 » n 

l<2<j<n 


{N + M + i + j){N - M - 1 + i + j) 
{i + jy 


(7.21) 


Proof. We set c = {N + M)/2, d — {N — M)/2, Xi = 1, i = 1, 2,..., n, in (3.14) 
and we set c = {N — M — 1)/2, d = {N + M — l)/2, Xi = 1, i = 1, 2,..., n, and 
replace p by n — p in (3.13). For this choice of xfs, a symplectic character reduces to 
a number (which is the dimension of the corresponding irreducible representation of 
Sp(2n, C)) which has a nice closed form (see [6, Ex. 24.20; 39, Theorem 4.5.(1)]), and 
is therefore easily computed. Thus, the left-hand sides of (3.14) and (3.13) turn into 
(7.19) and (7.20), respectively. On the right-hand sides of (3.14) and (3.13), we have 
certain sums of symplectic characters evaluated at Xi = 1, i — 1,2,... ,n. In the same 
way as in the proof of Theorem 4, these sums can be interpreted as the cardinalities 
of certain sets of plane partitions of trapezoidal shape. These are exactly the plane 
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partitions in the first assertion of Theorem 7, they are connted by the specialized 
right-hand side of (3.14) if -|- M is even, and by the specialized right-hand side of 
(3.13) if -|- M is odd. This proves the hrst assertion of Theorem 7. 

To obtain (7.21), we snm the respective expressions in (7.19) and (7.20) by means of 
Knmmer’s very well-poised 2^1 [—1] snmmation (cf. [35, (2.3.2.9), Appendix (III.5)]). 
Thns, Theorem 7 is proved. □ 

It wonld be easy to generalize the above theorem to trace generating fnnctions of 
the type appearing in [31, Theorem 1; 17, sec. 5] by replacing Xi by or in 

(3.13) and (3.14). We omit the details here. 

Appendix 

Al. Partitions and their diagrams. In Section 2 we already dehned a partition 
to be a seqnence A = (Ai, A 2 ,..., \r) of integers snch that Ai > A 2 > ■ ■ ■ > A^ > 0. A 
partition can be viewed geometrically, in terms of its Ferrers diagram. The Ferrers 
diagram of a partition A = (Ai, A 2 ,..., Xr) is an array of cells with r left-jnstihed 
rows and Xi cells in row i. Fignre A.a shows the Ferrers diagram corresponding to 
(4, 3, 3,1). We identify partitions with their Ferrers diagram. For example, if we say 
“the hrst row of the partition A” then we mean “the hrst row of the Ferrers diagram of 
the partition A.” The conjugate of a partition A is the partition A' = (A^, X' 2 , ■ ■ ■ ,X'^ ), 
where A' is the length of the j-th colnmn in the Ferrers diagram of A. 



a. Ferrers diagram b. skew Ferrers diagrams c. (4, 3, 3, l)/(2,1) d. (5, 4, 4, 2)/(3, 2,1,1) 

Figure A 

Given two partitions, half-partitions, or orthogonal (half-)partitions A = (Ai,A 2 , 
...) and fi = (//i, /U 2 ,...), we write A -|- // for the vector snm (Ai + /Ui, A 2 + /U 2 ,...). 
We write /U C A if < A^ for all i. Given two partitions A, fi with /U C A, the 
skew Ferrers diagram X/fx consists of all cells that are contained in (the Ferrers di¬ 
agram of) A bnt not in (the Ferrers diagram of) fi. Fignre A.b shows the skew 
Ferrers diagram (4, 3, 3, l)/(2,1). Of conrse, it also shows the skew Ferrers diagram 
(5,4, 4, 2)/(3, 2,1,1). So, the notation for skew Ferrers diagrams is not nniqne. We 
might even allow vectors containing negative coordinates for denoting skew Ferrers di¬ 
agrams, e.g., (3, 2, 2, 0)/(1, 0, —1, —1) for the same skew Ferrers diagram in Fignre A.b, 
or vectors with half-integer coordinates, e.g., (9/2, 7/2, 7/2, 3/2)/(5/2, 3/2,1/2,1/2) 
for the same skew Ferrers diagram. We do all this freely in the text, withont fnr- 
ther notice. However, when we connt colnmns of skew Ferrers diagrams X/fj. we do 
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distinguish between the different notations for the same skew Ferrers diagram. Each 
column gets the number that it has as a column of A. Thus, the first cell in the 
second row of the skew Ferrers diagram in Figure A.b is located in the second column 
of (4, 3, 3, l)/(2,1), see Figure A.c, in the third column of (5, 4,4, 2)/(3, 2,1,1), see 
Figure A.d, in the first column of (3, 2, 2, 0)/(l, 0, —1, —1), in the 5/2-th column of 
(9/2, 7/2, 7/2, 3/2)/(5/2, 3/2,1/2,1/2), etc. A horizontal strip is a skew Ferrers dia¬ 
gram with no more than one cell in each of its columns. A vertical strip is a skew 
Ferrers diagram with no more than one cell in each of its rows. 

A2. n-tableaux (ordinary tableaux). Let A,/U be partitions with // C A. An n- 
tableau of shape A, respectively of shape X/fr, is a filling of the cells of A, respectively 
A//U, with integers between 1 and n such that entries along rows are weakly increasing 
and entries along columns are strictly increasing. If we just say tableau instead of n- 
tableau then we mean the same but without requiring that the entries are bounded 
above by n. Figure 4 shows a 6-tableau (7-tableau, ... ) of shape (8, 8, 5, 3, 2). 

The weight x^ for an n-tableau T is defined by 

— #(l’s in T) #(2’s in T) #(n’s in T) / A n 

The vector (#(l’s in T), #(2’s in T),..., in T)) of exponents in (A.l) is called 

the content of T and is denoted by con(T). For example, the weight and content of 
the tableau in Figure 4 are x\x 2 x\x\x^xl and (4, 4, 5, 6, 5, 2), respectively. 

As is well-known (see [26, (5.12) with = 0; 34, Def. 4.4.1]), the irreducible 
general linear character (Schur function) Sn(A;x) equals the generating function for 
all n-tableaux of shape A, 


Sn(A;x)= ^ x'^, (A.2) 

T an n-tableau 
of shape A 

with x^ as defined in (A.l). 

A3. (2n)-symplectic tableaux. Let A = (Ai, A 2 ,..., A^) be a partition. A (2n)- 
symplectic tableau of shape A is a (2n)-tableau of shape 2A = (2Ai, 2 A 2 ,..., 2Xn) such 
that columns 1, 2, columns 3, 4, ... , columns 2Ai — 1, 2Ai form {2n)-symplectic 
admissible pairs. 

Definition 1. A pair {C,D) of two columns of the length k < n is called a (2n)- 
symplectic admissible pair if the following conditions are satisfied: 

(a) Entries in C and D are between 1 and 2n and in strictly increasing order. 

(b) If e is in D then 2n -t- 1 — e is not in D. The same holds for C. 

(c) C arises from D by a (possibly empty) sequence of operations O of the fol¬ 
lowing type: The operation O to be described applies only to columns E and 
integers ei, 62 with 62 < ei < n, where ei E E, 2n + 1 — €2 E E, and for all t 
between 62 and ei either t or 2n + 1 — t belongs to E. The operation O itself 
consists of forming the new column 0{E) out of E by replacing e\ by 62 and 
2 n -|- 1 — 62 by 2n -|- 1 — 61 and rearranging the new set of entries in strictly 
increasing order. 



48 


C. KRATTENTHALER 


For example, Figure 2 displays a 12-symplectic tableau of shape (4,4, 4,4, 3, 3). 
There, the next-to-last column arises from the last by one operation, as described in 
item (c) of the Dehnition, with e\ = 4, 62 = 3. 

Remark. Our description of (2n)-symplectic admissible pairs follows Lakshmibai [21], 
The description in [24, Appendix A.2] is equivalent. Note that Littelmann’s rows are 
our columns. 

There are a few observations that are immediate from the Dehnition. 

Observation 1. C < meaning that if the entries of C are ci, C 2 ,..., Cfc (from top 
to bottom) and those of D are di,d 2 ,. ■ ■ ,dk (from top to bottom), then Ci < di for 
all i. 

Observation 2. If (O', D) is a (2n)-symplectic admissible pair then the number of 
entries < n in (7 is the same as that in D. 

Observation 3. If C and D are columns of length n which satisfy (a), (b) in the 
Dehnition, C < D, and have the same number of entries < n, then (C, D) is a 
( 2 n)-symplectic admissible pair, i.e., condition (c) is satished automatically in this 
situation. This is due to the fact that a column of length n must contain either t or 
2n + 1 — t, for all t = 1, 2,..., n. Hence, the most obvious way to transform D into 
(7 is a legal sequence of operations according to (c): Namely, apply the operation O 
described in (c) hrst with ei the topmost entry of C and 62 the topmost entry of D 
(unless Cl = 62 ), then apply O with ei the next-to-the top entry of C and 62 the 
next-to-the-top entry of D, etc. 

The weight (x=*=^)'^ for a (2n)-symplectic tableau S is dehned by 

_ 4 (#(l’s in S)-#((2n)’s in S)) 4 (#(2’s in S)-#((2n-l)’s in S)) 

(x j Xi X 2 

4 (#(n’s in S)-#((n+l)’s in S)) / a o', 

Xu • A.O J 

Again, the vector 

^ (#(l’s in S) - #((2n)’s in A), #(2’s in S) - #((2n - l)’s in A), 

..., #(n’s in S) — #((n + l)’s in A)) 

of exponents in (A.3) is called the content of S and is denoted by con(5'). For example, 
the weight and content of the tableau in Figure 2 are x^x'^x^x^'^ and (0, 0,1, 2,1, —2), 
respectively. 

It is a theorem (see [3, 4, 23]) that the irreducible symplectic character sp 2 ni^', x=*=^) 
equals the generating function for all (2n)-symplectic tableaux of shape A, 

Aft„(A;x±‘)= ^ (x±')», (A.4) 

S a ( 2 n)-symplectic tableau 
of shape A 


with (x=*=^)'^ as dehned in (A.3). 
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A4. (2n + l)-orthogonal tableaux. Let A = (Ai, A 2 ,..., A^) be a partition or 

half-partition. A (2n -|- 1)-orthogonal tableau of shape A is a (2n)-tablean of shape 
2A = (2Ai, 2 A 2 ,..., 2An) snch that colnmns 2Ai — 1, 2Ai, colnmns 2Ai — 3, 2Ai — 2, 

... , form (2n -|- l)-orthogonal admissible pairs. 

Definition 2. A pair {C, D) of two columns of the length k < n is called a (2n + 1)- 
orthogonal admissible pair if the following conditions are satished: 

(a) Entries in C and D are between 1 and 2n and in strictly increasing order. 

(b) If e is in D then 2n -|- 1 — e is not in D. The same holds for C. 

(c) C arises from D by a (possibly empty) sequence of operations that can be 
either operations of the type that are described in Dehnition l.(c), or opera¬ 
tions O of the following type: The operation O to be described applies only 
to columns E and an entry e of E, e > n, where for all t between n -|- 1 and 
e, n -|- 1 included, either t or 2n -\-l — t belongs to E. The operation O itself 
consists of forming the new column 0{E) out of E by replacing e by 2n -t-1 — e 
and rearranging the new set of entries in strictly increasing order. 

For example, Figure 11 displays a 13-orthogonal tableau of shape (7/2, 7/2, 7/2, 
7/2, 7/2,1/2). There, the 4-th column arises from the 5-th by one operation as in item 
(c) of the Dehnition above, with e = 7, and by one operation as in Dehnition l.(c), 
with ei = 2 , 62 = 1 . 

Remark. Again, our description of (2n -|- l)-orthogonal admissible pairs follows Lak- 
shmibai [21]. The description in [24, Appendix, S'pm 2 m-i-i"Standard Young tableaux] 
is equivalent. Again, note that Littelmann’s rows are our columns. 

We make similar observations here, also immediate from the Dehnition. 

Observation 1. C < D, meaning that if the entries of C are ci,C 2 ,... ,Ck (from top 
to bottom) and those of D are di,d 2 ,... ,dk (from top to bottom), then Ci < di for 
all i. 

Observation 2. If C and D are columns of length n which satisfy (a), (b) in the 
Dehnition, and C < D, then {C,D) is a (2n -|- l)-orthogonal admissible pair, i.e., 
condition (c) is satished automatically in this situation. 

The weight (x=*=^)‘^ for a (2n -|- l)-orthogonal tableau S is again dehned by (A.3). 
Also here, the vector of exponents in (A.3) is called the content of S and is denoted 
by con(5'). 

It is a theorem (see [23]) that the irreducible orthogonal character S 02 n +i(A;x=^^) 
equals the generating function for all (2n -|- l)-orthogonal tableaux of shape A, 

s02n+i(A;x^^) = (x^^)^, (A.5) 

S a (2n-|-l)-orthogonal tableau 
of shape A 


with (x=*=^)‘^ as dehned in (A.3). 
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A5. (2n)-orthogonal tableaux. The definition of (2n)-orthogonal tableanx is the 
most intricate one. We provide the fnll definition for the sake of completeness. How¬ 
ever, in the cnrrent paper we actnally need only a special case, the one that is discnssed 
in Observation 1 below. The reader who is only interested in this paper’s applications 
of (2n)-orthogonal tableanx can safely skip the fnll definition and move on directly 
to Observation 1. 

We basically reprodnce Littelmann’s description [24, Appendix, A.3], with a little 
modification in the description of (2n)-orthogonal admissible pairs, where we follow 
[21]. Note again that Littelmann’s rows are onr columns. 

Let A = (Ai, A 2 ,..., An) be an n-orthogonal partition or half-partition. A (2n)- 
orthogonal tableau of shape A is a triple {Ss, S 2 , Si) of (2n)-tableanx of respective 
shapes ((An-i + A^)'"), ((An-i - A^)'"), and (2Ai - 2An-i, 2 A 2 - 2An-i,..., 2An-2 - 
2An-i), snbject to conditions (0)-(4) below. Note that when S 3 , S 2 , and Si are 
glned together, in that order, an array of shape (2Ai, 2 A 2 ,..., 2An-2, 2An-i, 2An-i) 
is obtained. 

(0) If e is an element of a colnmn of 5'2, or Ai then 2n -t- 1 — e is not an element 
of the colnmn. 

(1) Colnmns 1, 2, colnmns 3, 4, ... , colnmns 2Ai — 2An-i — 1, 2Ai — 2An-i of 
Si form (2n)-orthogonal admissible pairs (see the Definition below). In addition, for 
t = 1, 2,..., Ai — An-i — 1, let the entries of the (2t)-th colnmn of Si be (from top 
to bottom) ki, k 2 , ■ ■ ■, ks and let the entries of the { 2 i + l)-st colnmn be (again, from 
top to bottom) li,l 2 , ■ ■ ■ ,lt, s < t. Then for all seqnences 1 < ji < ■ ■ ■ < jq < s with 

n + 1 — q < kj.^ < ■ ■ ■ < kj^ <n + q 

and 

n -t- 1 — g < /jj < " ' < Ijq <n + q 

one has /jji + ■ ■ ■ + kj^ = + ■ ■ ■ + mod 2. (Note that this condition is empty if 

neither n nor n -|- 1 is an entry in one of the colnmns.) 

(2) The nnmber of entries > n in each colnmn of S 2 is odd. In addition, denote 
by R the colnmn (of length n) that arises from the leftmost colnmn of Si by adding 
all integers e, n -|- 1 < e < 2n, that together with their “conjngate” 2n -t- 1 — e do 
not already appear in the leftmost colnmn of Si, arranging everything in strictly 
increasing order, and replacing the smallest added element, / say, by 2n -|- 1 — / 
in case that the nnmber of entries > n wonld be odd otherwise. Denote by x the 
element of R that is closest to n -|- 1/2. To be precise, let ki, k 2 , ■ ■ ■, kg be the entries 
(from top to bottom) of the leftmost colnmn of Si. Then R consists of the entries 
ki,... ,ka,li,..., In-s-ij X with the following properties: 

2 n > h > ■ ■ ■ > In-s-i > n, R-s-i > x, R-s-i > 2 n + 1 — x, U ^ kj and 
X ^ kj for all 1 < i < n — s — 1, 1 < j < s, and if e E R then 2n + f — e ^ R. 
Fnrthermore, if the nnmber of integers strictly greater than n in is 

odd then x > n else x < n. 

Let R' be the colnmn with entries (i?\{a:}) U {2n -|- 1 — x} in increasing order. Then 
the concatenation S 2 U R' forms a (2n)-tablean. By the concatenation S 2 U R' we 
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mean that R' is glued from the right to S2 such that the topmost entries in each 
column are aligned in one row. 

( 3 ) The number of entries > n in each column of is even. Denote by S2 the 
tableau of shape ((An_i —An + 1 )"^) obtained from S2 as follows. The rightmost column 
of S2 is R. Now assume that 1 < i < A^-i — A^ and the (A^_i — A^ + 2 — i)-th column 
of S2 has already been dehned. The (A^-i — A^ + 1 — i)-th column of S2 consists 
of the entries of the (A^-i — A^^ + 1 — i)-th column of S2 with one entry e replaced 
by 2 n + 1 — e, arranged in strictly increasing order. The entry e to be chosen is the 
smallest possible such that rows will be weakly increasing (so that S2 indeed becomes 
a tableau). Then the concatenation 5*3 U is a ( 2 n)-tableau. By the concatenation 
S's U S2 we mean that is glued to the left to S2 such that the topmost entries in 
each column are aligned in one row. 

Definition 3 . A pair {C,D) of two columns of the length k < n is called a ( 2 n)- 
orthogonal admissible pair if the following conditions are satished: 

(a) Entries in C and D are between 1 and 2 n and in strictly increasing order. 

(b) If e is in D then 2 n + 1 — e is not in D. The same holds for C. 

(c) C arises from D by a (possibly empty) sequence of operations that can be 
either operations of the type that are described in Dehnition l.(c), or opera¬ 
tions O of the following type: The operation O to be described applies only to 
columns E and entries ei, 62 of i?, n < ci < 62, where for all t between n -|- 1 
and 62, n -|- 1 included, either t or 2 n + 1 — t belongs to E. The operation O 
itself consists of forming the new column 0 {E) out of E by replacing e\ by 
2 n -|- 1 — 61 and 62 by 2n -|- 1 — 62, and rearranging the new set of entries in 
strictly increasing order. 

Observation 1. (2n)-orthogonal tableaux of shape (A^-i, • ■ •, A^-i, A„) are just pairs 
(33,82) of (2n)-tableaux of respective shapes ((A^-i + A^)"') and ((A„_i — A^)"') 
such that each column of S3 or S2 does not contain 2n -|- 1 — 6 if it contains 6, 
such that the number of entries > n in S2 is odd and each entry in the hrst row 
of S2 is < n, such that the number of entries > n in S3 is even, and where the 
concatenation S3 U is a ( 2 n)-tableau, with S2 the tableau that arises from S2 by 
replacing the topmost element, 6j say, in column i of S2 by its “conjugate” 2n-l- 1 — 6^, 
for all i = 1 , 2 ,..., A^-i — A^, and by rearranging the columns in increasing order. 
For, in case of the above particular shape the tableau Si is empty, hence R equals 
{n -|- 1 , n -|- 2,..., 2n} or {n, n -|- 2,..., 2n}, depending on whether n is even or odd. 
Thus, R does not restrict S2 in item ( 2 ) above, except that for odd n it forces all the 
entries in the hrst row to be at most n. If n is even then all the entries in the hrst 
row have to be < n as well. For, in each column the number of entries > n is odd, 
with n being even. This implies that the number of entries < n has to be odd as well, 
in particular, it has to be at least 1 . Finally, in item ( 3 ) above, when forming S2 out 
of S2 always the smallest element in each column can be replaced by its “conjugate”. 

The weight (x^*^^)*^ for a ( 2 n)-orthogonal tableau S is again dehned by (A. 3 ). Also 
here, the vector of exponents in (A. 3 ) is called the content of S and is denoted by 
con( 5 '). 
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It is a theorem (see [23]) that the irreducible orthogonal character so 2 n(A; 
equals the generating function for all (2n)-orthogonal tableaux of shape A, 

s02n(A;x^^)= (A.6) 

S a (2n)-orthogonal tableau 
of shape A 


with (x^*^^)*^ as dehned in (A.3). 

A6. Littelmann’s decomposition rule and the Littlewood—Richardson rule. | 

Littelmann’s rule [24] for the decomposition of the product of two general linear, 
two symplectic, or two special orthogonal characters can be stated uniformly. (In 
fact, it is also valid for the simple, simply connected algebraic groups of type G 2 and 
Eq.) What is needed in the formulation of Littelmann’s theorem is the notion of 
(dominant) Weyl chamber associated to each of the characters. 

The (dominant) Weyl chamber of type A, which is associated to s„(.; x), is the set 
of points (cf. [6, p. 215]) 

{(xi, X 2 ,..., Xn) : Xi > X 2 > ■ ■ ■ > x^}. (A.7) 

The (dominant) Weyl chamber of type (7, which is associated to sp 2 n(-;x^^), is 
the set of points (cf. [6, p. 243, (16.5)]) 

{(Xi, X2, . . . , Xn) Xi > X2 > ■ ■ ■ > Xn > 0}. (A.8) 

The (dominant) Weyl chamber of type R, which is associated to S 02 n+i( ■; x=*^^), is 
the same set (A.8) of points (cf. [6, p. 272]). 

Finally, the (dominant) Weyl chamber of type D, which is associated to 
S 02 n( ■ ;x^^), is the set of points (cf. [6, p. 272]) 

{(xi,X 2 , ■■■,Xn) :xi>X 2 >---> \xn\}. (A.9) 

Furthermore, if T is a tableau, then by T(i) we denote the tableau that consists 
of the last £ columns of T. 

Now we are in the position to state Littelmann’s theorem. 

Theorem (Littelmann [24, Theorem, (a), p. 346]). Let Xn(-) be any of the char¬ 
acters Sn(. ;x), sp 2 n(-;x^^), S 02 n+i( • ;x=^^), or S 02 n( ■x^^). Then 

Xn(A) ■ Xn(h) = X] Xn(A + con(T)), (A.IO) 

T 

where the sum is over all corresponding tableaux (that is, n-tableaux in case Xn(-) — 
Sn(.;x), (2n)-symplectic tableaux in case Xn( ■) = sp 2 n( ■',x^^), etc.) of shape n such 
that A + con(T(£)) is in the Weyl chamber of the corresponding type for all 1. 

In case Xn( •) — Sn( •; x) this rule translates to the classical Littlewood-Richardson 
rule (cf. [26, I, sec. 9; 34, (4.26) + Theorem 4.9.4]), which we want to describe next. 
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Definition 4. A Littlewood-Richardson filling (LR-filling) of shape v/\ and content 
fi is an (ordinary) tablean F of shape n/X and content fi where the Littlewood- 
Richardson condition {LR-condition) is satisfied. The latter condition is the following 

Read the entries of F row-wise from top to bottom and in each row from 
right to left. Then at any stage dnring the reading the nnmber of I’s is 
greater or eqnal the nnmber of 2’s, which in tnrn is greater or eqnal the 
nnmber of 3’s, etc. 

Fnrthermore, define the Littlewood-Richardson coefficient by 

LR^ ^ = nnmber of LR-fillings of shape u/X and content fi. (A.11) 

Then the Littlewood-Richardson rule reads as follows, 

Sn(A; x) ■ Snin; x) = ^ LR^^^ Snin; x). (A.12) 

The translation from (A. 10) with Xn( ■) = 'Sn( •; x) fo (A. 12) proceeds as follows. 
It is basically the same idea as the one we nse in the proof of Proposition 1. What 
we do is to constrnct, for any fixed A, //, z/, a bijection between n-tableanx T of shape 
with u = X + con(T), and where A -|- con(T(£)) is in the Weyl chamber (A.7) of 
type A for all £, and LR-fillings F of shape u/X and content pi. Clearly, this wonld 
establish the eqnivalence of (A. 10) with Xn( ■) = 'Sn( ■; x) and (A. 12). 

Given T as above, we constrnct a seqnence Fq, Fi,..., F^^ of fillings by reading T 
colnmn-wise, from right to left. The desired filling F will then be defined to be the 
last filling, F^j. Define Fq to be the only filling of the shape A/A (which is of conrse 
the empty filling). Snppose that we already constrncted F^. To obtain F^_|_i, we add 
for i = 1,2, ...,n an entry e to row i of Fi if i is an entry occnring in the Fth last 
colnmn and the e-th row of T. As already annonnced, we define F to be F^^. 

A7. Littlewood’s branching rules. Here we quote the branching rules for the 
restriction of Schur functions to symplectic or orthogonal characters, which we use in 
Section 4. 

Theorem (Littlewood). There holds (see [25, App., p. 295; 13, (3.8b); 39, Theo¬ 
rem 3.13] 

s(A;x) = ^sp(z/;x) LR^,^ (A.13) 

ly even 

{‘F even’ means that all the columns of u are even), and (see [25, p. 240, (II); 13, 
(3.8a); 39, Theorem 3.16]; 

s(A;x) = ^o(z/;x) LR^,^^ (A.14) 

ly M? M 6ven 

(fi even’ means that all the rows of n are even), with LR^ the Littlewood-Richardson 
coefficients as defined in (A. 11). 
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